一种引入内存平衡的Hadoop平台作业调度算法

发布时间：2018-12-20 11:18

【摘要】：通过实验对FIFO、CAP和FAIR三种调度算法的特点与性能进行了对比分析,得出了它们各自优点和存在的问题.针对公平调度算法Fair Scheduler不适用于内存密集型作业调度的缺点,提出一种基于内存平衡的公平调度算法FMScheduler,在整个调度的过程中考虑作业的内存使用和计算节点的内存情况,通过加入内存比较机制、调整作业公平权重计算方法以及引入作业预留机制,对原有Hadoop公平调度算法进行改进与优化.最后,通过仿真实验对FMScheduler进行测试分析,实验结果表明,FMScheduler在高内存作业调度环境下的独立响应时间和作业整体的平均响应时间都比Fair Scheduler有所减少;并且在多用户多作业且包含内存密集型作业的环境中,FMScheduler与Hadoop原有的三种调度算法相比,在处理数据密集型作业和内存密集型作业的混合场景时,能够更合理公平地调度作业.
[Abstract]:The characteristics and performance of FIFO,CAP and FAIR scheduling algorithms are compared and analyzed through experiments, and their respective advantages and problems are obtained. Aiming at the disadvantage that fair scheduling algorithm Fair Scheduler is not suitable for memory-intensive job scheduling, a fair scheduling algorithm based on memory balance (FMScheduler,) is proposed to consider the memory usage of jobs and the memory situation of nodes in the whole scheduling process. By adding the memory comparison mechanism, adjusting the calculation method of job fair weight and introducing the job reservation mechanism, the original Hadoop fair scheduling algorithm is improved and optimized. Finally, the FMScheduler is tested and analyzed by simulation experiments. The experimental results show that the independent response time of FMScheduler and the average response time of the whole job in high memory job scheduling environment are lower than that of Fair Scheduler. Moreover, in the environment of multi-user and multi-job and including memory-intensive jobs, FMScheduler can schedule jobs more reasonably and fairly than the three original scheduling algorithms of Hadoop when dealing with the mixed scenarios of data-intensive jobs and memory-intensive jobs.
【作者单位】：海南大学信息科学技术学院;
【基金】：国家自然科学基金项目(71161007)资助海南省国际科技合作专项项目(KJHZ2014-16)资助海南省重点科技计划项目(ZDXM20130078)资助
【分类号】：TP333;TP301.6

【参考文献】