基于负载均衡的Hadoop平台下作业调度算法研究

发布时间：2018-06-10 01:16

本文选题：大数据 + MapReduce　；参考：《新疆大学》2013年硕士论文

【摘要】：随着信息技术的不断发展，各个企业的IT系统中存储着越来越多与企业运营息息相关的数据，可以说这些数据是企业发展的核心。所有IT系统的发展都依赖于数据。各行各业每天都会产生海量的数据，并且数据量的增长成爆炸式。2010年底，，据互联网数据中心的统计，全球数据量已经达到了120万PB，到2020年底，全球所有以电子形式存储的数据量将达到35ZB。人们不禁感叹，大数据时代已经到来。而Hadoop正是应大数据时代的到来而出现的。Hadoop是一个能够对海量数据进行分布式处理的软件框架。它是以一种可靠、高效、可伸缩的方式进行处理的。调度器（Scheduler）是Hadoop平台中一个非常重要的组件，它的主要功能是将系统中的空闲资源按照一定的策略分配给各个作业，它对于整个系统计算资源分配及作业执行起着至关重要的作用。因此我们对Hadoop作业调度器及算法的研究有着重要意义。本文首先介绍了Hadoop平台的优势及体系结构，接着对Hadoop的核心技术即Hadoop的分布式文件系统HDFS和Hadoop分布式数据处理MapReduce做了较为详细的介绍。然后对Hadoop原有的调度算法及LATE调度算法的原理及优缺点做了分析。另外，针对LATE调度算法在为落后任务选择备份执行节点策略的不足，本文提出了改进的LATE调度算法。该算法通过对Hadoop集群中的工作负荷进行分类，并提出对节点工作负载进行衡量的方法，进而在LATE调度算法的基础上提出了新的为落后任务选择备份执行节点的策略。最后介绍了拥有6个节点的Hadoop集群环境的搭建过程，并在该Hadoop集群上对LATE调度算法和改进的LATE调度算法进行了对比实验。实验结果表明，改进的LATE调度算法具有一定的优势。
[Abstract]:With the development of information technology, more and more data are stored in the IT system of each enterprise, which is the core of enterprise development. The development of all IT systems depends on data. All walks of life produce massive amounts of data every day, and the amount of data is exploding. By the end of 2010, according to the Internet data Center, the global data volume has reached 1.2 million PBs, and by the end of 2020, All data stored electronically around the world will reach 35 ZB. People can not help but sigh, big data era has come. Hadoop is a software framework that can deal with massive data distributed. It is handled in a reliable, efficient and scalable manner. Scheduler is a very important component in Hadoop platform. Its main function is to allocate the free resources to each job according to a certain policy. It plays an important role in computing resource allocation and job execution in the whole system. Therefore, it is of great significance to study Hadoop job scheduler and algorithm. Firstly, this paper introduces the advantages and architecture of Hadoop platform. Then, the core technology of Hadoop, namely, the distributed file system HDFS and Hadoop distributed data processing MapReduce, is introduced in detail. Then, the principle, advantages and disadvantages of Hadoop's original scheduling algorithm and path scheduling algorithm are analyzed. In addition, aiming at the deficiency of path scheduling algorithm in selecting backup execution node policy for backward tasks, this paper proposes an improved path scheduling algorithm. The algorithm classifies the workload in Hadoop cluster, and proposes a method to measure the workload of nodes, and then proposes a new strategy of selecting backup execution nodes for backward tasks based on the path scheduling algorithm. Finally, the construction process of Hadoop cluster environment with six nodes is introduced, and the comparison between the path scheduling algorithm and the improved path scheduling algorithm is carried out on the Hadoop cluster. Experimental results show that the improved path scheduling algorithm has some advantages.
【学位授予单位】：新疆大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP338.8

【相似文献】