基于大数据的Hadoop并行计算优化处理性能分析

发布时间：2019-03-28 12:24

【摘要】：随着近年来新一代移动通信、物联网、云计算等新兴技术的发展和普及，数据流量呈现爆发式增长，原有通信系统中的数据处理压力增大。而分布式计算Hadoop中的MapReduce编程架构凭借其强大的数据处理能力已经在文本分析、自然语言处理、商业数据处理等领域内成为比较成熟的解决方案，，能够满足现阶段通信领域的数据处理需求。但随着数据量的进一步增大、涉及的配置参数进一步增加，配置参数优化成为目前限制MapReduce性能的主要瓶颈。Hadoop配置涉及了200多个参数，其中有约13左右会对运行的作业产生较大的影响，这些参数恰恰决定了集群整体的性能表现。围绕上述问题，本文以性能调优为基础，设计一种新型的参数配置分析系统，能够针对每一个作业得到相应的最优化的参数配置。本文在原有MapReduce框架的基础上提出了三个新的逻辑部件：参数分析器、参数判决引擎、基于代价的优化模型。其中，参数分析器会在未修改的MapReduce程序中收集相关统计信息；参数判决引擎会对各个参数域进行细化的预测；基于代价的优化模型则是在前两者的基础上将参数配置简单化，给出最优化参数。上述三个部件综合使用下，可以针对每一个作业给出最优化的参数配置。通过对MapReduce主要领域内的典型应用：词频统计、词共现统计、排序进行优化后参数的综合评估来验证本文提出的三个逻辑部件的有效性。经过测试和验证，经过本文所设计的三个逻辑部件得出的最优化参数相比较经验法则与默认参数在解决Hadoop服务器集群的性能瓶颈问题上具有优越性和有效性。新的优化模型将参数配置简单化，且具有创新性和实用性。
[Abstract]:With the development and popularization of new generation mobile communication, Internet of things, cloud computing and other new technologies in recent years, the data flow increases explosively, and the pressure of data processing in the original communication system increases. The MapReduce programming architecture in distributed computing Hadoop has become a mature solution in the fields of text analysis, natural language processing, business data processing and so on with its powerful data processing ability. It can meet the requirement of data processing in the field of communication. However, as the amount of data increases, the configuration parameters involved further increase, configuration parameters optimization has become the main bottleneck limiting the performance of MapReduce. Hadoop configuration involves more than 200 parameters, About 13 of them will have a great impact on the running jobs, and these parameters determine the performance of the cluster as a whole. Based on performance tuning, a new type of parameter configuration analysis system is designed in this paper, which can optimize the configuration of parameters for each job. In this paper, based on the original MapReduce framework, three new logical components are proposed: parameter analyzer, parameter decision engine and cost-based optimization model. Among them, the parameter analyzer will collect the relevant statistics in the unmodified MapReduce program, and the parameter decision engine will refine the prediction of each parameter domain. The cost-based optimization model simplifies the configuration of parameters on the basis of the first two and gives the optimal parameters. Under the combined use of the above three components, the optimal parameter configuration can be given for each job. The validity of the three logic components proposed in this paper is verified by the comprehensive evaluation of the optimized parameters of the typical applications in the main fields of MapReduce: word frequency statistics, word co-occurrence statistics and ordering. After testing and verification, compared with the optimal parameters obtained by the three logical components designed in this paper, the rule of thumb and the default parameters are superior and effective in solving the bottleneck problem of Hadoop server cluster performance. The new optimization model simplifies the configuration of parameters and is innovative and practical.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP274.2;TN92

【相似文献】