面向多核可伸缩的MapReduce库的研究

发布时间：2019-01-28 20:22

【摘要】：并行编程是有效利用多核资源的重要方式。然而,传统的并行编程模型要求程序员手动管理诸多的并发细节,使得并行编程变得困难且易错。MapReduce编程模型通过自动管理并发任务,简化并行编程。Phoenix是MapReduce模型在多核系统上的一种实现,它证明了应用程序通过使用MapReduce能够获得与Pthreads相媲美的性能。但是,Phoenix的可伸缩性较差。本文旨在分析影响Phoenix可伸缩性的关键因素,然后设计并实现一个具有较好可伸缩性的MapReduce库SMR。本文的主要贡献为:(1)通过实验测试以及使用Linux Perf等性能分析工具,深入分析影响Phoenix可伸缩性与性能的关键因素。Phoenix因Pthreads线程对共享地址空间的竞争导致其较差的可伸缩性,因map和reduce阶段间存在同步屏障影响性能。(2)利用课题组已有的新型线程模型Sthread,设计并实现一个Phoenix接口兼容且具有较好可伸缩性的MapReduce库SMR。SMR通过避免多个线程对共享地址空间的竞争,提升系统的可伸缩性;通过让map和reduce阶段间的无阻塞流水线并行,提升系统的性能。(3)在一台32核机器上使用Phoenix测试程序对SMR和Phoenix进行性能评估,还分析了近年来Linux内核的优化对多核MapReduce库可伸缩性与性能的影响。实验结果表明,基于SMR运行的五个测试用例都具有较好的可伸缩性和性能。32核CPU下,SMR相比Phoenix的性能提升为9.0倍～26.7倍。Linux内核的优化能够有效提升Phoenix的执行效率,而对于SMR的影响较小。(4)为了高效支持迭代型MapReduce应用,本文初步设计并实现了面向多核可伸缩的迭代式MapReduce库iSMR。SMR因每次迭代计算都需要重复地创建和销毁环境,浪费大量的时间,致使其无法高效地支持迭代型应用。iSMR只在处理流程的开始和结束时创建和销毁环境,并且它采用线程池复用线程,以提升迭代型应用的处理效率。本文提出的SMR为程序员提供了一种简单高效的方式使用多核资源。此外,本文关于多核MapReduce库可伸缩性的分析,对其他Pthreads程序的分析具有指导意义;关于近年来Linux内核优化的分析,对Linux的相关研究工作具有参考价值。
[Abstract]:Parallel programming is an important way to utilize multi-core resources effectively. However, the traditional parallel programming model requires programmers to manage many concurrency details manually, which makes parallel programming difficult and error-prone. The MapReduce programming model manages concurrent tasks automatically. Simplified parallel programming. Phoenix is an implementation of the MapReduce model on multi-core systems, which proves that applications can achieve performance comparable to Pthreads by using MapReduce. However, Phoenix has poor scalability. The purpose of this article is to analyze the key factors that affect the scalability of Phoenix, and then design and implement a MapReduce library SMR. with good scalability. The main contributions of this paper are as follows: (1) through experimental testing and using Linux Perf and other performance analysis tools, Deeply analyze the key factors that affect the scalability and performance of Phoenix. Phoenix has poor scalability due to the competition between Pthreads threads and shared address space. Due to the existence of synchronization barrier between map and reduce phase, the performance is affected. (2) using Sthread, a new thread model, which has been developed by our research group. A Phoenix interface compatible and scalable MapReduce library SMR.SMR is designed and implemented to enhance the scalability of the system by avoiding the competition of multiple threads on the shared address space. The performance of the system is improved by parallel non-blocking pipeline between map and reduce. (3) using Phoenix test program to evaluate the performance of SMR and Phoenix on a 32-core machine. The effects of Linux kernel optimization on the scalability and performance of multi-core MapReduce libraries are also analyzed. The experimental results show that the five test cases based on SMR have good scalability and performance. Under 32 core CPU, the performance of SMR is 9.0 ~ 26.7 times higher than that of Phoenix. The optimization of Linux kernel can effectively improve the execution efficiency of Phoenix. In order to support the iterative MapReduce application efficiently, this paper designs and implements a multi-core scalable iterative MapReduce library iSMR.SMR, which needs to create and destroy the environment repeatedly for each iteration calculation. ISMR only creates and destroys the environment at the beginning and end of the processing process, and it uses thread pool to reuse threads to improve the processing efficiency of iterative applications. The proposed SMR provides programmers with a simple and efficient way to use multicore resources. In addition, the analysis of the scalability of multi-core MapReduce libraries is of guiding significance to the analysis of other Pthreads programs, and the analysis of Linux kernel optimization in recent years has a reference value for the research work of Linux.
【学位授予单位】：中国科学技术大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.1

【相似文献】