基于hadoop的任务资源分配与控制

发布时间：2018-04-23 21:42

本文选题：hadoop + 接纳控制　；参考：《北京邮电大学》2016年硕士论文

【摘要】：Hadoop是一个开发和运行处理大规模数据的软件平台,是Apache的一个用java语言实现开源软件框架,实现在大量计算机组成的集群中对海量数据进行分布式计算。Hadoop框架中最核心设计就是:HDFS和MapReduce。HDFS提供了海量数据的存储,MapReduce提供了对数据的计算。Hadoop作为云计算的一种解决方案也越来越受到人们的重视。Hadoop资源调度器的为需要计算机资源的任务分配资源,使他们能够顺利完成计算任务。好的资源调度可以充分利用资源,规避任务运行中的问题,提高计算机资源利用率。因此如何做好hadoop的任务资源分配与控制具有重要的意义。本文在广泛研究调度算法的基础上,利用作业执行的历史数据来指导hadoop的任务资源分配与控制。本文首先介绍了一个信息采集系统。信息采集系统能够采集、传输、保存计算节点的实时信息,map任务和reduce任务的执行信息,job执行信息。我们将采集的信息来保存在数据库中供后面调度使用。在广泛细致的研究公平调度的基础上,发现了公平调度的两个能够改进的地方。首先,每个任务的内存需求不同。程序运行都是需要一定的内存支持的,如果一个任务需要相当大的内存,但是被分配到执行任务的计算节点不能够提供足够的内存,使导致任务不能够正常的执行。那么这个任务会在该计算节点上执行相当缓慢,影响其他任务的执行。其次,公平调度算法是通过任务的个数来保证节点的负载均衡。但是由于每个任务都有自己的特点,占用的资源不同、作业的类型不同。这样分配不能达到很好的负载均衡。本文提出利用监控控的历史数据估算即将调度的作业的内存和当前节点内存状况,预计作业是否能够顺利完成,从而进行接纳控制。通过分析任务队列中作业类型和节点上任务的类型,从作业队列中选择最优的任务,达到负载均衡。
[Abstract]:Hadoop is a software platform for developing and running large-scale data processing. It is an open source software framework of Apache using java language. The core design of distributed computing. Hadoop framework for mass data in a cluster of computers is that: HDFS and MapReduce.HDFS provide storage of mass data MapReduce provides a solution for computing data. Hadoop is a solution to cloud computing. More and more people pay attention to the solution. Hadoop resource scheduler allocates resources for tasks that need computer resources. Make it possible for them to complete the task of calculation. Good resource scheduling can make full use of resources, avoid problems in task operation, and improve the utilization rate of computer resources. Therefore, how to do a good job of hadoop task resource allocation and control has important significance. Based on the extensive study of scheduling algorithms, this paper uses the historical data of job execution to guide the task resource allocation and control of hadoop. This paper first introduces an information collection system. The information collection system can collect, transmit and save the real-time information of the computing node and the execution information of the reduce task. We store the collected information in the database for later scheduling. On the basis of extensive and detailed research on fair scheduling, two improvements of fair scheduling are found. First, the memory requirements for each task are different. If a task needs a considerable amount of memory, the computing node allocated to the task can not provide enough memory, which results in the task being unable to execute normally. This task will be executed slowly on the compute node, affecting the execution of other tasks. Secondly, the fair scheduling algorithm ensures the load balance of nodes by the number of tasks. However, each task has its own characteristics, occupying different resources and different types of jobs. This allocation does not achieve good load balance. In this paper, we propose to estimate the memory status of the job to be scheduled and the memory status of the current node by using the historical data of the monitor and control, and to estimate whether the job can be successfully completed, so as to carry out admission control. By analyzing the job type in the task queue and the task type on the node, the optimal task is selected from the job queue to achieve load balancing.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【参考文献】