MALK:一种高效处理大规模键值的MapReduce框架
发布时间：2018-11-09 21:24
【摘要】：内存申请是引发共享存储系统上MapReduce性能下降的主要瓶颈之一,特别是对于需要处理大量键值的应用尤为严重.为了解决此问题,提出了一种内存开销低、能高效处理大规模键值的MapReduce并行计算框架——MALK(high-efficient MapReduce for applications having large amount of keys).MALK对于离散的大规模键值采用连续的存储管理方法,避免了大量小块内存的申请;通过更细粒度地处理Map阶段的任务和流水化Reduce阶段的任务,来减少系统运行过程中同时活跃的数据量,从而将应用程序对内存的需求控制在一个较小的范围内;并提出一种Hash表的复用机制,通过复用Hash表的存储空间来避免流水过程中Hash表内存的重复申请;MALK还综合考虑了任务的粒度和数量对任务管理开销和整体性能的影响,把Reduce阶段的任务数量设成对系统性能最优的值.实验结果表明:相对于Phoenix++,MALK的性能最高可提升3.8倍(平均2.8倍);在Map和Reduce阶段,MALK最多可节省95.2%和87.8%的存储空间;MALK在Reduce阶段还取得了更好的负载均衡,降低了L2和LLC Cache的缺失率.
[Abstract]:Memory request is one of the main bottlenecks that lead to the deterioration of MapReduce performance on shared storage systems, especially for applications that need to deal with a large number of keys and values. In order to solve this problem, a MapReduce parallel computing framework, MALK (high-efficient MapReduce for applications having large amount of keys). MALK), is proposed, which has low memory overhead and can deal with large scale key values efficiently. MALK (high-efficient MapReduce for applications having large amount of keys). MALK) uses a continuous storage management method for discrete large scale key values. Avoid a large number of small blocks of memory applications; In order to reduce the amount of data active in the running process of the system by handling the tasks in the Map phase and the pipelined Reduce phase in a finer granularity, the requirements of the application program for memory are kept within a relatively small range. A reuse mechanism of Hash table is proposed to avoid the repeated request of Hash table memory in pipeline process by multiplexing the storage space of Hash table. MALK also considers the effects of the granularity and number of tasks on the task management overhead and overall performance comprehensively, and sets the number of tasks in the Reduce phase as the optimal value for system performance. The experimental results show that compared with Phoenix, MALK, the performance of MALK can be increased by 3.8 times (average 2.8 times), and the storage space of MALK can be saved by 95.2% and 87.8% at the stage of Map and Reduce. MALK also achieved better load balance in the Reduce phase, reducing the missing rate of L2 and LLC Cache.
【作者单位】： 计算机体系结构国家重点实验室(中国科学院计算技术研究所);中国科学院大学;首都师范大学信息工程学院;
【基金】：国家“九七三”重点基础研究发展计划基金项目(2011CB302501) 国家杰出青年科学基金项目(60925009) 国家自然科学基金项目(60921002,61173007,61100013,61100015,61202059,61202055) 国家“八六三”高技术研究发展计划基金项目(2012AA012301,2012AA010303) 北京市科技新星计划基金项目(2010B058) 计算机体系结构国家重点实验室开放课题(CARCH201203)
【分类号】：TP333
[Abstract]:Memory request is one of the main bottlenecks that lead to the deterioration of MapReduce performance on shared storage systems, especially for applications that need to deal with a large number of keys and values. In order to solve this problem, a MapReduce parallel computing framework, MALK (high-efficient MapReduce for applications having large amount of keys). MALK), is proposed, which has low memory overhead and can deal with large scale key values efficiently. MALK (high-efficient MapReduce for applications having large amount of keys). MALK) uses a continuous storage management method for discrete large scale key values. Avoid a large number of small blocks of memory applications; In order to reduce the amount of data active in the running process of the system by handling the tasks in the Map phase and the pipelined Reduce phase in a finer granularity, the requirements of the application program for memory are kept within a relatively small range. A reuse mechanism of Hash table is proposed to avoid the repeated request of Hash table memory in pipeline process by multiplexing the storage space of Hash table. MALK also considers the effects of the granularity and number of tasks on the task management overhead and overall performance comprehensively, and sets the number of tasks in the Reduce phase as the optimal value for system performance. The experimental results show that compared with Phoenix, MALK, the performance of MALK can be increased by 3.8 times (average 2.8 times), and the storage space of MALK can be saved by 95.2% and 87.8% at the stage of Map and Reduce. MALK also achieved better load balance in the Reduce phase, reducing the missing rate of L2 and LLC Cache.
【作者单位】： 计算机体系结构国家重点实验室(中国科学院计算技术研究所);中国科学院大学;首都师范大学信息工程学院;
【基金】：国家“九七三”重点基础研究发展计划基金项目(2011CB302501) 国家杰出青年科学基金项目(60925009) 国家自然科学基金项目(60921002,61173007,61100013,61100015,61202059,61202055) 国家“八六三”高技术研究发展计划基金项目(2012AA012301,2012AA010303) 北京市科技新星计划基金项目(2010B058) 计算机体系结构国家重点实验室开放课题(CARCH201203)
【分类号】：TP333
【参考文献】
相关期刊论文 前4条
1 张书彬;韩冀中;刘志勇;王凯;;基于MapReduce实现空间查询的研究[J];高技术通讯;2010年07期
2 王珊;王会举;覃雄派;周p，
本文编号：2321597
本文链接：https://www.wllwen.com/kejilunwen/jisuanjikexuelunwen/2321597.html
