Hadoop中作业的自适应资源调度策略研究与实现

发布时间：2018-04-02 19:58

本文选题：异构集群　切入点：计算能力　出处：《华中科技大学》2016年硕士论文

【摘要】：随着基于Hadoop平台的大数据技术不断发展和实践的深入,Hadoop YARN(Yet Anouther Resource Negotiator)资源调度策略在异构集群中的不适用性越发明显。一方面,YARN资源调度器无法根据节点的计算能力动态调整节点承担的任务份额,导致异构集群中优势节点的计算资源浪费、系统性能没有充分发挥;另一方面,现有的资源调度策略始终为作业静态地分配统一规格的资源容器,未考虑作业执行的不同阶段资源需求的差异性,易产生大量资源碎片,从而导致系统资源利用率降低,整体性能下降。基于以上问题,在深入分析YARN架构及其资源调度机制的基础上提出了作业的自适应资源调度策略:首先,监控服务器对集群所有执行节点和提交的作业进行多项性能相关信息的监控;其次,利用采集的实时监控数据建模、量化集群节点的综合计算能力;最后,集群主节点结合实时节点性能监控信息和作业性能监控信息启动基于相似度评估的动态资源调度方案。优化后的系统能够有效识别集群节点的执行能力差异,并根据作业任务的实时需求进行细粒度的动态资源调度,在完善YARN现有调度语义的同时,可作为子级资源调度方案架构在上层调度器下。搭建Hadoop2.0和Ganglia综合实验平台,对上述作业的自适应资源调度策略进行实现,并基于大数据典型CPU密集型作业和I/O密集型作业进行性能测试。实验结果表明,作业的自适应资源调度策略能够有效增加集群并发度、缩短作业执行时间、提升系统资源利用率。
[Abstract]:With the continuous development and practice of big data technology based on Hadoop platform, the inapplicability of Hadoop YARN(Yet Anouther Resource negotiator resource scheduling strategy in heterogeneous clusters becomes more and more obvious.On the one hand, the YARN resource scheduler can not dynamically adjust the task share of the node according to the computing power of the node, which leads to the waste of computing resources of the dominant node in the heterogeneous cluster, and the system performance is not given full play.The existing resource scheduling strategy always assigns uniform resource container to the job statically, and does not consider the difference of resource requirements in different stages of job execution, which is easy to produce a large number of resource fragments, which leads to the decrease of system resource utilization.Overall performance decline.Based on the above problems, an adaptive resource scheduling strategy for jobs is proposed on the basis of in-depth analysis of the YARN architecture and its resource scheduling mechanism.The monitoring server monitors the performance related information of all the execution nodes and the jobs submitted by the cluster. Secondly, using the collected real-time monitoring data modeling, quantifies the comprehensive computing ability of the cluster nodes.The dynamic resource scheduling scheme based on similarity evaluation is initiated by cluster master node combining real-time node performance monitoring information and job performance monitoring information.The optimized system can effectively identify the difference of execution ability of cluster nodes, and carry out fine-grained dynamic resource scheduling according to the real-time requirements of job tasks.It can be used as a sublevel resource scheduling scheme architecture under the upper scheduler.A comprehensive experimental platform of Hadoop2.0 and Ganglia was built to implement the adaptive resource scheduling strategy of the above jobs and to test the performance based on big data typical CPU intensive jobs and I / O intensive jobs.The experimental results show that the adaptive resource scheduling strategy can effectively increase the concurrency degree of the cluster, shorten the job execution time and improve the system resource utilization.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【参考文献】