基于HDFS默认副本选择机制的改进和实现

发布时间：2018-01-07 22:08

本文关键词：基于HDFS默认副本选择机制的改进和实现　出处：《北京交通大学》2017年硕士论文　论文类型：学位论文

更多相关文章： HDFS 副本选择 心跳机制 负载均衡

【摘要】：随着人类社会快速的发展,人们日常生活产生了海量数据,传统的数据处理方法已不适合海量数据的分析处理,Hadoop应运而生。Hadoop有两个最核心的组成部分,分别是MapReduce和HDFS。MapReduce具有处理分析海量数据的能力,HDFS具有存储和管理海量数据的能力。HDFS的副本选择机制直接影响着数据的可靠性、可用性、均衡性和读取效率。由于HDFS默认副本选择机制选择副本的存放位置具有一定的随机性且未充分考虑数据节点的负载状况,进而出现HDFS数据分布不均匀和Hadoop集群负载不均衡等问题,本文对HDFS的默认副本选择机制进行改进和实现,主要包含以下方面:首先,本文提出根据数据节点当前的CPU使用率、内存使用率、磁盘IO使用率、磁盘使用率以及带宽使用率五个因素综合描述其在集群中的负载状况,这五个因素为数据节点负载状况的量化打下了基础。根据每种因素对数据节点负载影响的大小赋予其不同的权重值,量化数据节点负载状况的数值。其次,本文剖析DataNode到NameNode周期性心跳机制的原理,改进的副本选择机制利用心跳机制将影响数据节点负载的因素汇报给NameNode,使NameNode掌握整个集群数据节点的负载状况。通过对HDFS默认副本选择机制的分析,针对其存在的缺陷,根据每个数据节点、机架的负载状况以及集群中所有数据节点的平均负载状况,提出了对HDFS默认副本选择机制的改进方法。最后,编译修改后的HDFS源代码,搭建Hadoop集群环境,以本文改进的HDFS副本选择机制分别对低于三副本、三副本和大于三副本进行实验。经实验结果分析,表明改进的HDFS副本选择机制能够依据数据节点的负载状况选择最佳的副本存放位置,保证了数据的可靠性、可用性,均衡了数据在集群中的分布,提高了数据读写速度,改善了集群的负载均衡。
[Abstract]:With the rapid development of human society, the massive data of daily life, the traditional data processing methods is not suitable for the analysis of massive data processing, Hadoop.Hadoop comes with two of the most important core part, namely MapReduce and HDFS.MapReduce has the processing capability of huge amounts of data, a copy of HDFS has the ability to store and manage the massive data the.HDFS selection mechanism directly affects the reliability of data, availability, balance and reading efficiency. Because HDFS is the default copy selection mechanism selects the location with the copy machine and did not fully consider the load status of certain data nodes, and HDFS data distribution and Hadoop load imbalance problem, this paper the HDFS default replica selection mechanism for the improvement and implementation, mainly includes the following aspects: firstly, this paper according to the node The current CPU usage, memory usage, disk IO usage, disk usage and bandwidth utilization rate of five factors described in the cluster load conditions, the five factors for quantitative data node load of foundation. According to the influence of each factor on the size of the given node load data the different weights, numerical quantitative data of the node load. Secondly, this paper analyzes the principle of NameNode DataNode to the periodic heartbeat mechanism, improved replica selection mechanism using heartbeat mechanism will affect the reporting data to the NameNode node load factor, load condition to make NameNode master the entire cluster of data nodes. Through the analysis of the HDFS default replica selection the mechanism of the defects, according to each data node, the average load of all nodes and load data frame in the cluster, put forward to HDF The improved method of S default replica selection mechanism. Finally, the modified HDFS compiler source code, build Hadoop cluster environment, HDFS replica selection mechanism to the improved of less than three copies, three copies and more than three copies of the experiment. By the analysis of experimental results, show that the improved HDFS copy selection mechanism according to the data of nodes the load status of selecting the best replica location, to ensure the reliability of data, availability, balance the distribution of data in the cluster, improve the speed of reading and writing data, improve the cluster load balancing.

【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333;TP393.09

【相似文献】