基于GPU的Spark大数据技术在实验室的开发应用

发布时间：2018-05-27 01:16

  本文选题：大数据处理 + 异构计算　； 参考：《实验室研究与探索》2017年01期

【摘要】：在大数据时代,兼顾大数据处理与高性能计算是目前对计算机系统的迫切需求。针对Spark大数据处理与基于GPU的高性能计算,分析了基于GPU的Spark技术。它主要通过构建CPU和GPU的异构并行,使计算机获得强大的计算能力,并在实验室环境下探讨了Spark-GPU技术的实现,阐述了算法实现的技术流程。在此基础上,通过仿真实验评估了Spark和Spark-GPU技术的性能。实验表明,Spark-GPU技术可以达到上百倍的加速比,这对图像处理以及信息检索等领域的发展都具有重要推动作用。
[Abstract]:In the era of big data, it is an urgent need for computer system to combine big data processing with high performance computing. Aiming at Spark big data processing and high performance computing based on GPU, the Spark technology based on GPU is analyzed. By constructing the heterogeneous parallelism of CPU and GPU, the computer can obtain powerful computing power. The realization of Spark-GPU technology is discussed in the laboratory environment, and the technical flow of algorithm realization is expounded. On this basis, the performance of Spark and Spark-GPU technology is evaluated by simulation experiments. Experiments show that the Spark-GPU technology can achieve a speedup ratio of several hundred times, which plays an important role in the development of image processing and information retrieval.
【作者单位】： 南京信息工程大学电子与信息工程学院;
【基金】：国家自然科学基金(NSFC61203273) 江苏省自然科学基金(BK20141004)
【分类号】：TP311.13
，

本文编号：1939755