MapReduce大数据处理平台与算法研究进展

发布时间：2018-01-19 08:10

本文关键词： 大数据 MapReduce 外存算法大数据处理算法性能优化　出处：《软件学报》2017年03期 　论文类型：期刊论文

【摘要】：综述了近年来基于MapReduce编程模型的大数据处理平台与算法的研究进展.首先介绍了12个典型的基于MapReduce的大数据处理平台,分析对比它们的实现原理和适用场景,抽象其共性;随后介绍基于MapReduce的大数据分析算法,包括搜索算法、数据清洗/变换算法、聚集算法、连接算法、排序算法、偏好查询、最优化算法、图算法、数据挖掘算法,将这些算法按照MapReduce实现方式分类,分析影响算法性能的因素;最后,将大数据处理算法抽象为外存算法,并对外存算法的特征加以梳理,提出了普适的外存算法性能优化方法的研究思路和问题,以供研究人员参考.具体包括优化外存算法的磁盘I/O、优化外存算法的局部性以及设计增量式迭代算法.现有的大数据处理平台和算法研究多集中在基于资源分配和任务调度的平台动态性能优化、特定算法并行化、特定算法性能优化等领域,所提出的外存算法性能优化属于静态优化方法,是现有研究的良好补充,为研究人员提供了广阔的研究空间.
[Abstract]:This paper summarizes the research progress of big data processing platform and algorithm based on MapReduce programming model in recent years. Firstly, 12 typical big data processing platforms based on MapReduce are introduced. Analyze and compare their realization principle and applicable scene, abstract their commonness; Then the big data analysis algorithm based on MapReduce is introduced, including search algorithm, data cleaning / transformation algorithm, aggregation algorithm, join algorithm, sorting algorithm, preference query, optimization algorithm. Graph algorithm, data mining algorithm, these algorithms are classified according to the MapReduce implementation mode, and the factors that affect the performance of the algorithm are analyzed. Finally, the big data processing algorithm is abstracted as an external memory algorithm, and the characteristics of the external memory algorithm are combed, and the research ideas and problems of the universal external memory algorithm performance optimization method are put forward. For the reference of researchers. Including the optimization of the disk I / O memory algorithm. The existing big data processing platform and algorithms focus on the dynamic performance optimization of the platform based on resource allocation and task scheduling, and the parallelization of specific algorithms. The performance optimization of the external storage algorithm is a static optimization method, which is a good supplement to the existing research and provides a broad research space for researchers.
【作者单位】：东北大学软件学院;东北大学计算机科学与工程学院;
【基金】：国家自然科学基金(61672143,61433008,61402090,61502090)~~
【分类号】：TP311.13
【正文快照】： Research Advance on Map Reduce Based Big Data Processing Platforms and AlgorithmsSONG Jie1,SUN Zong-Zhe1,MAO Ke-Ming1,BAO Yu-Bin2,YU Ge21(Software College,Northeastern University,Shenyang 110819,China)2(School of Computer Science and Engineering,Northeas

【相似文献】