面向Hadoop存储系统的节能优化技术研究

发布时间：2018-05-10 13:51

本文选题：节能存储 + Hadoop系统　；参考：《华中科技大学》2012年硕士论文

【摘要】：近年来，基于云计算的互联网服务不断涌现，其中MapReduce计算范式和HDFS分布式文件系统已逐渐成为开发大型数据密集型应用的首选模型。从硬件供应商的角度，这类应用部署的规模如此巨大，降低服务集群的功率消耗既可以显著降低运营成本，又能降低碳排放量，从而提高整体能效。在传统服务器节能策略的基础上，针对提供MapReduce作业服务的集群，提出一种节能优化算法。该算法能根据集群当前整体和局部的工作负载动态地重构节点或节点上的数据；同时，控制数据放置策略很好的支持上述操作。该节能优化算法具备了节能效果明显、实时性高以及负载均衡开销小等特性，，可应用于数据密集型计算集群和企业数据中心等环境中。具体地，实现节能优化的能耗控制系统由数据分发模块、节点失效模块和能耗调节模块三个组件组成。数据分发模块通过对HDFS数据块分发和对应副本放置流程进行修改，实现人为控制数据块号到DataNode节点映射；节点失效模块使得HDFS具备容忍节点增加/缺失的功能；能耗调节器是提高能效的核心，包含两个线程，分别对应两种算法：dilution和enrichment。在集群的某个机架利用率高于管理员预定的阈值时，能耗调节模块会根据dilution算法适时地添加新节点，并将附近节点上的数据迁移到新节点上；当集群出现某个机架的利用率偏低时，能耗调节器可依据enrichment算法移除目标节点，并其上的数据迁移到附近节点上。从而，实现系统当前工作负载与系统性能的动态匹配。在最终测试方面，利用GridSim Toolkit对节能优化算法从功能和节能效果两个方面进行了测试和评估，前者主要验证了enrichment和dilution算法是否能在集群负载变化时重构节点或者数据；后者着重测试了在平均负载和低负载条件下节能的效果，并与传统Covering Set技术进行了比较。实验结果表明提出的节能优化算法在MapReduce计算中高负载情况下能达到30.32%的节能效果，在低负载情况下能达到69.77%的节能效果。
[Abstract]:In recent years, cloud based Internet services have springing up. The MapReduce computing paradigm and HDFS distributed file systems have gradually become the preferred model for developing large data intensive applications. From the point of view of hardware vendors, the scale of such applications is so large that the power consumption of lower service clusters can be significantly reduced. Operating costs can also reduce carbon emissions, thereby improving overall energy efficiency.
On the basis of the traditional server energy saving strategy, an energy-saving optimization algorithm is proposed for the cluster providing MapReduce job service. This algorithm can dynamically restructure the data on nodes or nodes according to the current overall and local workload of the cluster. At the same time, the control data placement strategy supports the above operation very well. The method has the characteristics of obvious energy saving effect, high real time and low load balance, which can be used in data intensive computing cluster and enterprise data center.
Specifically, the energy control system for energy saving optimization is composed of three components: data distribution module, node failure module and energy consumption regulation module. The data distribution module can control the data block number to the DataNode node by changing the HDFS data block distribution and corresponding replica placement process, and the node failure module makes HDFS The energy regulator is the core of increased energy efficiency; the energy regulator is the core of energy efficiency improvement, including two threads, which correspond to two algorithms respectively: dilution and enrichment. can add new nodes according to the dilution algorithm at a time when the utilization rate of a frame in the cluster is higher than that of the administrator. The data on the point is migrated to the new node. When the utilization rate of a frame is low, the energy regulator can remove the target nodes according to the enrichment algorithm and migrate the data to the nearby nodes. Thus, the current workload of the system can be matched with the dynamic state of the system performance.
In the final test, the GridSim Toolkit is used to test and evaluate the energy saving optimization algorithm from two aspects of function and energy efficiency. The former mainly validates whether the enrichment and dilution algorithms can reconstruct nodes or data when the cluster load changes; the latter focuses on testing the energy saving under the average load and low load conditions. The results are compared with the traditional Covering Set technology. The experimental results show that the proposed energy saving optimization algorithm can achieve 30.32% energy saving effect in the case of high load in the MapReduce calculation, and can achieve 69.77% energy saving effect under low load conditions.

【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【相似文献】