基于Hadoop集群的节能优化技术研究

发布时间：2018-06-14 12:27

本文选题：集群 + 节能优化　；参考：《华中科技大学》2013年硕士论文

【摘要】：计算机集群的广泛应用给人们带来方便的同时，也带来了能耗问题，如何有效地利用能源，降低不必要的能耗已成为一个迫切需要解决的关键问题。作为目前最热门的海量数据处理框架，Hadoop被部署到越来越多的集群上。但是Hadoop却一直没有引入节能的特性，因此对Hadoop集群的节能优化技术进行研究是非常有必要的。基于现有的研究成果，结合Hadoop本身的机架感知特性和副本存放策略，提出了根据集群的负载情况调整集群规模来改善集群能源利用率的节能方案。节能方案为集群添加两个功能模块：集群负载监控模块和集群规模调整模块。负载监控模块负责监控集群的利用率，当监控到利用率过高或过低时会通知集群规模调整模块对集群规模进行调整。集群规模调整模块包含两个重要的算法：退役节点算法和重用节点算法，集群利用率过低时会启动退役节点算法，通过退役节点来减少集群中活跃节点数量，，以提高集群能源利用率，集群利用率过高时会启动重用节点算法来增加集群活跃节点数量，以提高集群对任务请求的响应速度。通过使用GridSim工具包来对系统进行仿真实验，实验中对比了未使用节能策略的集群状态和应用节能策略的集群状态，验证了退役节点算法和重用节点算法确实能够根据集群的负载来改变集群的规模，实验结果表明在算法执行过程中可以使集群节能30%以上。
[Abstract]:The extensive application of computer cluster brings people convenience and energy consumption. How to use energy effectively and reduce unnecessary energy consumption has become a key problem that needs to be solved urgently. Hadoop, the most popular mass data processing framework, is deployed to more and more clusters. However, Hadoop has not introduced the characteristics of energy saving, so it is necessary to study the energy-saving optimization technology of Hadoop cluster. Based on the existing research results, combined with Hadoop's own rack perception characteristics and replica storage strategy, an energy saving scheme is proposed to improve cluster energy efficiency by adjusting cluster scale according to the load situation of the cluster. The energy saving scheme adds two functional modules to the cluster: the cluster load monitoring module and the cluster scale adjustment module. The load monitoring module is responsible for monitoring the utilization rate of the cluster. When monitoring the utilization rate is too high or too low, the cluster size adjustment module will be notified to adjust the cluster size. The cluster scale adjustment module includes two important algorithms: the decommissioned node algorithm and the reuse node algorithm. When the cluster utilization is too low, the decommissioned node algorithm is started, and the number of active nodes in the cluster is reduced by decommissioned nodes. In order to improve the cluster energy utilization, if the cluster utilization is too high, the reuse node algorithm will be started to increase the number of active nodes in the cluster, so as to improve the response speed of the cluster to the task request. By using the GridSim toolkit to simulate the system, the cluster state without energy saving strategy and the cluster state with energy saving strategy are compared in the experiment. It is verified that the retired node algorithm and the reuse node algorithm can change the scale of the cluster according to the load of the cluster. The experimental results show that the algorithm can save more than 30% energy in the process of implementing the algorithm.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】