Hadoop平台下调度算法及其改进策略研究

发布时间：2018-06-17 10:24

本文选题：Hadoop + MapReduce　；参考：《北京邮电大学》2016年硕士论文

【摘要】：当今社会随着信息技术发展,互联网的普及,每天都有大量与人们活动相关的数据产生。而数据产生的同时,如何能够有效的存储大量数据并对数据进行有效分析,成为了人们迫切需要解决的问题。与此同时,由于数据量的激增,诸如数据挖掘以及网页索引等数据敏感应用需要处理不断增长的从几GB到几TB甚至PB的数据集。针对上述问题,Google提出了 MapReduce的编程模型。MapReduce编程模型的主要思想是用户只需要表述想要执行的计算,而不必关心并行计算、容错、数据分布以及负载均衡等细节。Hadoop是MapReduce模型的开源实现之一。Hadoop框架包含两个主要部分:HDFS(Hadoop Distributed File System)和 MapReduce,HDFS用于海量数据的存储,MapReduce用于对海量数据进行研究分析。Hadoop因其高可靠性、高扩展性、高容错性以及低廉的成本,迅速成为了流行的大数据处理平台之一。本文通过文献调研,针对Hadoop分布式计算平台进行了研究。首先从Hadoop平台的产生入手,论述了 Hadoop平台产生的背景及意义,并且对Hadoop平台架构及关键技术进行了研究。其次,文中研究了 Hadoop平台现有的三种调度算法,即FIFO调度、计算能力调度(Capacity Scheduler)以及公平份额调度(Fair Scheduler),其中主要分析了各个调度算法产生意义、工作原理以及不足。然后本文基于现有三种调度算法的不足之处,提出了动态作业匹配调度算法(Dynamic Matching Based on Memory Scheduler,DMBMScheduler),新的算法在作业调度时考虑了内存数据局部性,并在作业调度时依据实时匹配原则来进行作业分配。最后对算法进行实现以及实验测试,实验结果表明我们的算法成功达到了预期的目标,缩短了作业执行时间以及作业响应时间,解决了现有算法的不足,并提高了 Hadoop平台的整体性能。
[Abstract]:With the development of information technology and the popularization of Internet, a lot of data about people's activities are produced every day. At the same time, how to store and analyze the data effectively becomes an urgent problem. At the same time, due to the rapid increase of data volume, data-sensitive applications such as data mining and web page indexing need to deal with growing data sets ranging from several gigabytes to several terabytes or even PB. The main idea of MapReduce programming model. MapReduce programming model is that users only need to express the calculation they want to perform, and do not care about parallel computing, fault-tolerant, etc. Data distribution and load balancing. Hadoop is one of the open source implementations of MapReduce model. The Hadoop framework consists of two main parts: HDFSU Hadoop distributed File system) and MapReduceHDFS for mass data storage. Hadoop is used to study and analyze mass data. Hadoop is highly reliable. High scalability, high fault tolerance and low cost have rapidly become one of the popular big data processing platforms. In this paper, the Hadoop distributed computing platform is studied through literature investigation. Firstly, the background and significance of Hadoop platform are discussed, and the architecture and key technologies of Hadoop platform are studied. Secondly, this paper studies three scheduling algorithms of Hadoop platform, namely FIFO scheduling, capacity scheduling capacity Scheduler and Fair Scheduler, in which the significance, working principle and deficiency of each scheduling algorithm are analyzed. Then, based on the shortcomings of the existing three scheduling algorithms, this paper proposes a dynamic matching based on memory scheduling algorithm. The new algorithm takes memory data locality into account in job scheduling. And in the job scheduling according to the principle of real-time matching to carry out job assignment. Finally, the algorithm is implemented and tested. The experimental results show that our algorithm has successfully achieved the desired goal, shortened the job execution time and job response time, and solved the shortcomings of the existing algorithms. The whole performance of Hadoop platform is improved.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】