MapReduce模型下数据本地性负载平衡策略研究

发布时间：2018-04-04 22:08

本文选题：云计算　切入点：MapReduce　出处：《大连海事大学》2014年硕士论文

【摘要】：随着社会的发展,互联网中的数据信息处于爆炸式的增长状态,每天互联网中都会产生海量的数据。可以想象,以后人们所面临的数据规模会越来越巨大。从这些巨大的数据中如何能够快速高效地寻找到有价值的数据资源变得越来越重要。正是在这样的背景下,云计算这个新型的数据处理技术正在快速的发展。 MapReduce是云计算技术中的一种并行处理大规模数据的处理模型,它具有简单易用、高可扩展性和高容错性的特点,因此被广泛应用于云计算领域。Hadoop平台是MapReduce编程模型一个具体的应用,目前许多公司和高校都采用它开发和研究云计算技术。然而,它的一些处理机制影响着它性能的发挥。当面对倾斜数据时,原有的分区算法无法均衡的分配数据到各处理节点,这样就导致了处理节点间负载不平衡,并且造成网路负载加重,从而导致集群性能的降低。本文所提出的方法针对MapReduce作业过程中的Partition部分的分区算法进行调整和优化,设计实现了一个基于数据本地性的负载均衡策略(DALP)。通过合理的抽样,对作业数据进行预处理,分析出数据中各key的频数分布情况,然后利用分析的数据制定一个数据聚合策略(DA),该策略考虑了key的数据大小,可以更加合理地为各Reduce节点分配数据,能有效地缓解集群节点的负载不平衡问题,提高了集群整体性能。此外,针对集群中网络带宽制约集群性能的问题,结合对平台数据本地性的深入研究,又提出了一个基于数据本地性的数据分配策略(LP),在数据聚合的基础上利用本地性数据分配方法,能够有效减少集群间数据流动情况,减少网络负载。本文所提出的方法通过实验得到了充分验证,数据倾斜时的负载问题得到了有效地改善。
[Abstract]:With the development of the society, the data information in the Internet is in the explosive growth state, every day in the Internet will produce a huge amount of data.One can imagine that the scale of data people will face in the future will grow larger and larger.How to find valuable data resources quickly and efficiently from these huge data becomes more and more important.It is against this background that cloud computing, a new data processing technology, is developing rapidly.MapReduce is a parallel processing model for large-scale data processing in cloud computing technology. It has the characteristics of easy to use, high scalability and high fault tolerance.Therefore, it is widely used in cloud computing. Hadoop platform is a concrete application of MapReduce programming model. At present, many companies and universities use it to develop and research cloud computing technology.However, some of its processing mechanisms affect its performance.When the data is tilted, the original partition algorithm can not distribute the data to each processing node evenly, which leads to the imbalance of the load between the processing nodes and the increase of the network load, which leads to the degradation of the performance of the cluster.The method proposed in this paper adjusts and optimizes the partition algorithm of Partition in the process of MapReduce operation, and designs and implements a load balancing strategy based on data nativeness.Through reasonable sampling, preprocessing the job data, analyzing the frequency distribution of each key in the data, then using the analyzed data to make a data aggregation strategy, this strategy considers the data size of key.The data can be allocated to each Reduce node more reasonably, which can effectively alleviate the load imbalance of cluster nodes and improve the overall performance of the cluster.In addition, aiming at the problem that the network bandwidth restricts the performance of the cluster, combined with the in-depth research on the platform data localization,Furthermore, a data allocation strategy based on data locality is proposed. Based on data aggregation, using local data allocation method can effectively reduce data flow among clusters and reduce network load.The proposed method is fully verified by experiments, and the load problem when the data is tilted is improved effectively.
【学位授予单位】：大连海事大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.09

【相似文献】