HDFS异构集群中的分级存储调度机制

发布时间：2018-09-02 07:42

【摘要】：大部分存储集群构建时可能包含有遗留设备及新购置设备,这些设备在存储性能方面存在较大差异.采用HDFS默认的机架感知存储策略时,可能使访问频率高的数据存储在低性能节点上,而访问频率低的数据存储在高性能节点上,既影响集群响应时间,又降低了资源利用率.针对以上问题,提出一种分级存储调度机制.在HDFS机架感知调度策略基础上,首先根据节点的CPU、内存大小、磁盘大小、磁盘I/O等固有硬件性能将节点划分为高配置节点和低配置节点,其次根据节点的CPU使用率、内存使用率、网络带宽使用率、磁盘使用率等性能的动态因素建立节点的性能评价模型,并建立三个性能级别.根据节点配置情况、性能级别及网络位置等多方面因素进行综合调度.同时在集群运行过程中,会根据数据的访问频率对数据块的分布进行动态调整.实验结果表明,本文提出的分级存储调度机制可以在HDFS异构集群中提高数据的访问效率,优化集群性能.
[Abstract]:Most storage cluster builds may include legacy devices and newly acquired devices, which differ greatly in storage performance. When the HDFS default rack aware storage strategy is adopted, the data with high access frequency may be stored on the low performance node, while the data with low access frequency will be stored on the high performance node, which will not only affect the cluster response time, but also reduce the resource utilization rate. Aiming at the above problems, a hierarchical storage scheduling mechanism is proposed. Based on the HDFS rack aware scheduling strategy, the nodes are divided into high configuration nodes and low configuration nodes according to the CPU, memory size of the node and the inherent hardware performance of disk I / O, and then the CPU utilization rate of the nodes. The dynamic factors of memory utilization, network bandwidth utilization, disk utilization and other performance factors establish the performance evaluation model of nodes and establish three performance levels. According to the node configuration, performance level and network location and other factors for comprehensive scheduling. At the same time, the data block distribution will be dynamically adjusted according to the data access frequency during the running process of the cluster. Experimental results show that the proposed hierarchical storage scheduling mechanism can improve data access efficiency and optimize cluster performance in HDFS heterogeneous clusters.
【作者单位】：北方工业大学云计算研究中心;大规模流数据集成与分析技术北京市重点实验室;
【基金】：北京市教育委员会科技计划重点项目(KZ201310009009)资助北京市属高等学校创新团队建设与教师职业发展计划基金项目(IDHT20130502)资助
【分类号】：TP333

【参考文献】