MapReduce模型的性能优化研究

发布时间：2018-05-06 22:43

本文选题：MapReduce + 动态调度　；参考：《郑州大学》2017年硕士论文

【摘要】：随着互联网、云计算以及物联网的快速发展,电子商务、电子政务、社交网络等新应用为人们的日常生活和工作带来极大方便,同时也使数据产生的方式越来越多样化,数据量呈爆炸式增长。在大数据的时代,MapReduce以其高效率、易扩展、简易性等一系列特性,成为现阶段海量数据处理的主流模型。但是,MapReduce现有的数据分配机制易导致输入数据倾斜的问题,造成少数的几个点上分配了大部分的数据,最终导致各节点的负载不同;在现实生活中需要被处理的海量数据大部分都是呈偏态分布得,即Zipf分布,这样就会导致一些数据对应的记录数不均等。同时,易导致分区一样的数据汇聚到性能低的节点上,造成各节点作业执行时间不同的现象。对于密集型数据任务,在拉取数据时会造成大量的磁盘访问以及竞争有限的网络宽带资源等瓶颈。MapReduce性能优化的关键问题之一是数据倾斜。为了优化MapReduce数据倾斜的问题,在本文中提出了MapReduce在线抽样分区的负载均衡优化机制。该机制在任务开始之前,首先对源数据进行抽样分析操作,来预测源数据分布的特征;根据数据分布特征,动态地调用不同的数据分区优化策略;在任务执行过程中,实时监测每个节点的负载,同时动态优化对应的数据分区策略。为提升异构环境下的MapReduce性能,本文提出一种异构环境下基于节点作业时间感知的动态MapReduce调度策略:DTHE(Dynamic MapReduce scheduling based on the Time-aware of node jobs in Heterogeneous Environments)。DTHE在作业执行前,首先标记部分任务作为节点样本任务并优先处理,在执行其他任务时分析样本任务,预测节点性能和数据分布特征,动态采取相应的调度策略;在作业运行中实时监测节点任务状态,提前拉取节点下一个任务数据到本地内存。实验结果表明:在异构环境下,DTEH能够缩短5.1%的作业执行时间并减少磁盘I/O,有效提升MapReduce性能。
[Abstract]:With the rapid development of the Internet, cloud computing and the Internet of things, new applications such as e-commerce, e-government, social networking bring great convenience to people's daily life and work, and make the way of data generation more and more diverse. The amount of data increased explosively. In big data's time, MapReduce has become the mainstream model of mass data processing because of its high efficiency, expansibility and simplicity. However, the existing data distribution mechanism of MapReduce can easily lead to the problem of input data skew, resulting in the distribution of most of the data on a few points, resulting in different load of each node. In real life, most of the massive data that need to be processed are skewed distribution, that is, Zipf distribution, which will lead to some data corresponding to the number of records is not equal. At the same time, it is easy to converge the same data into the low performance nodes, resulting in different job execution time of each node. For the intensive data tasks, data skew is one of the key problems in the performance optimization of MapReduce, which will cause a lot of disk access and limited network broadband resources. In order to optimize the skew of MapReduce data, a load balancing optimization mechanism for MapReduce online sampling partition is proposed in this paper. Before the task starts, the mechanism first carries on the sampling analysis to the source data to predict the source data distribution characteristic, according to the data distribution characteristic, dynamically invokes different data partition optimization strategy, in the task execution process, The load of each node is monitored in real time and the corresponding data partition strategy is dynamically optimized. In order to improve the performance of MapReduce in heterogeneous environment, this paper proposes a dynamic MapReduce scheduling strategy based on Node Job time Awareness: MapReduce dynamic MapReduce scheduling based on the Time-aware of node jobs in Heterogeneous Environments).DTHE before job execution. First, some tasks are labeled as node sample tasks and processed first, then the sample tasks are analyzed when other tasks are executed, and the node performance and data distribution characteristics are predicted, and corresponding scheduling strategies are adopted dynamically. The task state of the node is monitored in real time during the operation, and the next task data is pulled to the local memory in advance. The experimental results show that DTEH can shorten the job execution time by 5.1% and reduce the disk I / O in heterogeneous environment, which can effectively improve the performance of MapReduce.
【学位授予单位】：郑州大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【相似文献】