基于Hadoop系统的自学习资源调度器模型研究

发布时间：2018-02-12 10:40

本文关键词： Hadoop 资源调度自学习 MapReduce 作业　出处：《华中科技大学》2016年硕士论文　论文类型：学位论文

【摘要】：随着信息大爆炸时代的来临,云计算和大数据技术应运而生。Hadoop是一个支持分布式集群使用简单的MapReduce编程模型处理大规模数据集的框架。当集群规模不断增长时,如何提高集群的资源利用率、缩短任务的响应时间,优化Hadoop的资源调度器,提高集群的效率,已成为当前云计算领域的研究热点。本文结合国内外的研究现状,在比较了Hadoop系统常见的几种资源调度器的基础上,改进了一种基于作业分类的自学习资源调度器模型,以期提高异构Hadoop集群的资源利用率,同时缩短作业的执行时间。本文主要完成了以下研究内容:第一,介绍了Hadoop系统的发展史和国内外有关Hadoop资源调度器的研究现状。第二,阐释了Hadoop系统的两大核心——分布式文件系统HDFS和并行编程模型MapReduce的原理。第三,详细分析了当前Hadoop系统的三种资源调度器:FIFO、Capacity Scheduler和Fair Scheduler,解释了它们的实现原理,分析了各自的优缺点和适用场景。第四,对Hadoop系统进行建模,每一个节点的资源可以抽象为虚拟核和内存,虚拟核有一个执行速率的属性,内存有两个属性,分别是大小和数据到达速率。系统有三个性能评价指标:本地特性、作业平均完成时间、公平性。第五,改进自学习资源调度器模型,并用实验验证。首先构建作业分类器,每类作业都有一个队列与之对应。当作业到来时,把作业加入相应类别的队列。自学习资源调度器在后台维护一个各类作业资源需求量的配额表,调度器根据历史统计数据采用特定的资源实时动态分配策略,定期更新配额表,形成正反馈调节。在实验阶段,选择单词统计、排序和矩阵相乘这三类作业做对比实验,比较了Hadoop系统分别使用FIFO调度器、计算能力调度器和自学习资源调度器三种情况下作业的完成时间、集群的CPU使用率和内存利用率,从而得出结论:自学习资源调度器在处理Reduce阶段计算量小且耗时短的作业和磁盘IO次数少、计算密集型的作业方面,能够显著地缩短作业完成时间并且提高集群的资源利用率。
[Abstract]:With the advent of the information explosion era, cloud computing and big data technology came into being. Hadoop is a framework that supports distributed clusters to process large data sets using a simple MapReduce programming model. How to improve cluster resource utilization, shorten task response time, optimize Hadoop resource scheduler and improve cluster efficiency has become the research hotspot in the field of cloud computing. On the basis of comparing several common resource schedulers in Hadoop system, a self-learning resource scheduler model based on job classification is improved to improve the resource utilization of heterogeneous Hadoop cluster. At the same time, the execution time of jobs is shortened. This paper mainly completes the following research contents: firstly, it introduces the history of Hadoop system and the research status of Hadoop resource scheduler at home and abroad. The principle of distributed file system (HDFS) and parallel programming model (MapReduce) are explained in this paper. Thirdly, three kinds of resource schedulers:: FIFO capacity Scheduler and Fair Scheduler of current Hadoop system are analyzed in detail, and their implementation principles are explained. This paper analyzes their advantages and disadvantages and applicable scenarios. 4th, the Hadoop system is modeled. The resources of each node can be abstracted into virtual core and memory. The virtual core has an attribute of execution rate, and memory has two attributes. The system has three performance evaluation indexes: local property, average job completion time, fairness. 5th. The model of self-learning resource scheduler is improved and verified by experiments. Firstly, a job classifier is constructed. Each type of job has a queue corresponding to it. When the job arrives, add the job to the queue of the corresponding class. The self-learning resource scheduler maintains a quota table for the requirements of each job resource in the background. According to the historical statistics, the scheduler adopts a specific real-time dynamic allocation strategy of resources, and periodically updates the quota table to form positive feedback adjustment. In the experiment stage, three kinds of jobs, namely, word statistics, sorting and matrix multiplication, are selected to do comparative experiments. In this paper, we compare the job completion time, CPU usage and memory utilization of Hadoop system using FIFO scheduler, computing power scheduler and self-learning resource scheduler, respectively. It is concluded that the self-learning resource scheduler can significantly shorten the job completion time and improve the resource utilization of the cluster in processing Reduce with less computation time and less disk IO times and computation-intensive jobs.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【参考文献】