基于Hadoop平台的分布式任务调度算法研究

发布时间：2018-10-30 17:37

【摘要】：随着信息技术的发展,互联网作为一种服务越来越深远地影响人类的生活。在信息爆炸的背景下,海量信息处理成为计算机科学的新挑战。MapReduce是一个并行分布式数据处理编程模型,它的优势在于简化传统分布式程序的开发,使得开发人员只需专注于业务逻辑程序的编写无需考虑分布式的实现细节。Hadoop是MapReduce的开源实现,它是众多的海量数据处理企业、研究机构提供数据处理的基础平台。MapReduce调度算法主要解决的是集群共享,集群资源的利用,作业的响应时间等问题。同时,随着用户的实时性要求的增加关于MapReduce的实时调度的研究也越来越多。MapReduce的实时调度的难点在于作业实时调度的模型,同时又要考虑到集群的异构性和数据的本地性。任务剩余时间预测是实时调度的重要部分,剩余时间的预测往往易受到集群的异构性特点的影响。通过研究Hadoop作业运行时机制,本文提出了一种自适应的Reduce任务调度算法(Self-Adaptive Reduce Scheduling, SARS)。在现有的MapReduce调度算法研究中,Reduce任务的调度时间过于简单。Reduce任务的调度时间直接影响Reduce的完成时间和集群资源的利用率。SARS调度算法能够根据作业自身的特性决定Reduce任务的调度时间。实验结果表明SARS调度算法减少了Reduce的完成时间和集群作业的平均响应时间,进一步提高了集群资源的利用率。通过研究集群的异构性,本文提出了一种基于计算能力的节点分类算法,用于划分集群中不同计算能力的节点。在节点分类算法的基础上,提出了一种基于时间约束的实时调度算法(MapReduce Task Scheduling for Deadline constraints, MTSD)。MTSD提出了一种任务剩余时间评估模型,进而推导出实时作业调度时资源的需求模型。通过实验表明MTSD提高了MapReduce作业的数据本地性,并能够完成实时调度需求。
[Abstract]:With the development of information technology, the Internet, as a service, has a profound impact on human life. Under the background of information explosion, mass information processing has become a new challenge in computer science. MapReduce is a parallel distributed data processing programming model, which has the advantage of simplifying the development of traditional distributed programs. Hadoop is an open source implementation of MapReduce, which is a large number of massive data processing enterprises. MapReduce scheduling algorithm mainly solves the problems of cluster sharing, utilization of cluster resources, response time of jobs and so on. At the same time, with the increase of real-time requirements of users, there are more and more researches on real-time scheduling of MapReduce. The difficulty of real-time scheduling of MapReduce lies in the model of real-time job scheduling, and the heterogeneity of cluster and the locality of data must be considered at the same time. Task residual time prediction is an important part of real-time scheduling, and the prediction of remaining time is often affected by the heterogeneity of cluster. This paper proposes an adaptive Reduce task scheduling algorithm (Self-Adaptive Reduce Scheduling, SARS).) by studying the mechanism of Hadoop job runtime. In the existing MapReduce scheduling algorithm research, The scheduling time of Reduce task is too simple. The scheduling time of Reduce task directly affects the completion time of Reduce and the utilization of cluster resources. SARS scheduling algorithm can decide the scheduling time of Reduce task according to the characteristics of the job itself. Experimental results show that the SARS scheduling algorithm reduces the completion time of Reduce and the average response time of cluster jobs, and further improves the utilization of cluster resources. By studying the heterogeneity of cluster, this paper proposes a node classification algorithm based on computing power, which is used to divide the nodes with different computing power in the cluster. On the basis of node classification algorithm, a real-time scheduling algorithm based on time constraint, (MapReduce Task Scheduling for Deadline constraints, MTSD). MTSD, is proposed, and a task residual time evaluation model is proposed. Then the resource requirement model of real-time job scheduling is derived. Experiments show that MTSD improves the data localization of MapReduce jobs and can fulfill the real-time scheduling requirements.
【学位授予单位】：湖南大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP338.8

【相似文献】