基于Hadoop的作业调度负载均衡算法研究

发布时间：2018-01-24 17:27

本文关键词： 云计算 Hadoop MapReduce LATE 负载均衡　出处：《安徽理工大学》2016年硕士论文　论文类型：学位论文

【摘要】：随着Web2.0时代的到来,互联网的迅速发展,人们对网络的依赖越来越多。尤其是“互联网+”的提出,各大企业均响应号召,实施转型,各行各业每天都能产生海量数据,且呈爆炸式增长。由于海量信息的存储需求和新的计算能力的需求,促使了新的计算模式-云计算的诞生。而Hadoop是云平台中一个能够对大数据进行分布式处理的开源平台,是MapReduce编程模型的开源实现,也是众多学者进行大数据研究的首选平台。Hadoop的一个重要组件就是其调度器,它主要实现作业的调度执行和系统资源的合理分配,其调度算法的优劣对集群的性能有着至关重要的作用。因此对Hadoop调度器及算法的研究有着重要意义。本文通过研究Hadoop平台的现有算法,分析LATE算法的调度原理及优劣,针对其在异构环境下选择备份任务及执行节点时的不足,提出一个改进的IR-LATE调度算法。该算法首先对集群中的作业负载进行分类,选择备份执行时,通过IR-LATE调度算法为剩余完成时间最长、最需要备份的慢任务启动备份并选择最优节点执行。最后通过实验验证IR-LATE算法与LATE算法相比,不仅提高了对慢任务判定的正确性,还缩短了作业的平均运行时间,提升了集群的负载均衡。
[Abstract]:With the arrival of the Web2.0 era and the rapid development of the Internet, people rely more and more on the network, especially the "Internet" proposed, all major enterprises respond to the call to implement the transformation. A variety of industries can produce huge amounts of data every day and grow explosively due to the storage requirements of mass information and the need for new computing power. Hadoop is an open source platform for distributed processing of big data in the cloud platform and an open source implementation of the MapReduce programming model. It is also an important component of big data research platform. Hadoop is its scheduler, which mainly realizes the job scheduling execution and the rational allocation of system resources. The advantages and disadvantages of its scheduling algorithm is very important to the performance of the cluster. Therefore, the research of Hadoop scheduler and algorithm is of great significance. This paper studies the existing algorithms of Hadoop platform. This paper analyzes the scheduling principle of LATE algorithm and its advantages and disadvantages, aiming at its shortcomings in selecting backup tasks and executing nodes in heterogeneous environment. An improved IR-LATE scheduling algorithm is proposed, which first classifies the job load in the cluster. When the backup execution is selected, the remaining completion time is the longest through the IR-LATE scheduling algorithm. The slow task that needs the backup most starts the backup and selects the optimal node to execute. Finally, the IR-LATE algorithm is proved to be more accurate than the LATE algorithm in comparison with the LATE algorithm. It also shortens the average running time of the job and improves the load balance of the cluster.
【学位授予单位】：安徽理工大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】