云计算中MapReduce的性能优化研究

发布时间：2018-07-05 16:00

本文选题：云计算 + MapReduce　；参考：《南京邮电大学》2017年硕士论文

【摘要】：随着信息技术的快速发展和普及应用,大规模的数据处理需求日益增加,传统的并行计算机在存储空间和计算资源方面难以满足海量数据处理的需求,因此云计算技术为解决海量数据处理提供了良好的环境。MapReduce编程模型为云计算进行海量数据处理提供了新的思路,它克服了传统分布式并行程序编写复杂的缺点,提供了简化的编写方式。但是MapReduce编程模型的性能上仍然存在一些问题,因此本文从任务调度的角度对MapReduce的性能进行优化。本文研究的主要内容是:首先对云计算和Hadoop平台进行简单介绍,重点研究MapReduce计算模型和任务调度算法,在分析MapReduce调度算法不足的基础上,提出了蚁群模拟退火算法,在该算法中,以减少任务的完成时间和保证资源负载均衡为目标,根据蚁群算法构造局部最优解,将蚁群算法的局部最优解作为模拟退火算法的初始解进行全局搜索,并根据Metropolis准则判断是否接受当前解。针对MapReduce编程模型中的容错技术的缺点,本文提出引入失效恢复机制的可靠性任务调度策略,对云环境中的资源节点进行可信任度评估,建立可信任度模型,根据可信任度模型为任务分配可靠的节点,避免任务失败重新分配执行,浪费时间和资源。最后,通过仿真平台CloudSim验证了本文提出的任务调度算法和调度模型的有效性和稳定性。
[Abstract]:With the rapid development and widespread application of information technology, the demand for large-scale data processing is increasing day by day. Traditional parallel computers are unable to meet the requirements of mass data processing in storage space and computing resources. Therefore, cloud computing technology provides a good environment for mass data processing. MapReduce programming model provides a new way for cloud computing to process mass data. It overcomes the complex shortcomings of traditional distributed parallel programs. A simplified way of writing is provided. However, there are still some problems in the performance of MapReduce programming model, so this paper optimizes the performance of MapReduce from the perspective of task scheduling. The main contents of this paper are as follows: firstly, the cloud computing and Hadoop platforms are briefly introduced, and the MapReduce computing model and task scheduling algorithm are studied. Based on the analysis of the lack of MapReduce scheduling algorithm, an ant colony simulated annealing algorithm is proposed. In order to reduce the completion time of the task and ensure the resource load balance, the local optimal solution of the ant colony algorithm is constructed according to the ant colony algorithm, and the local optimal solution of the ant colony algorithm is considered as the initial solution of the simulated annealing algorithm. The Metropolis criterion is used to determine whether the current solution is acceptable or not. Aiming at the shortcomings of fault-tolerant technology in MapReduce programming model, this paper proposes a reliability task scheduling strategy based on failure recovery mechanism, evaluates the trust degree of resource nodes in cloud environment, and establishes a trust degree model. A reliable node is assigned to the task according to the trust model to avoid the task failure redistribution and waste of time and resources. Finally, the effectiveness and stability of the proposed task scheduling algorithm and scheduling model are verified by the simulation platform CloudSim.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP3;TP18

【参考文献】