基于内存优化配置的MapReduce性能调优

发布时间：2019-01-23 09:10

【摘要】：MapReduce作业性能与内存配置存在极大的相关性,针对准确预测作业内存困难问题,根据Java虚拟机(JVM)的分代内存管理特点,提出了一种分代内存预测方法.首先使用回归模型对年轻代与垃圾回收平均时间的关系进行建模,将寻找合理年轻代内存大小的问题转换为一个受约束的非线性优化问题,并设计搜索算法来求解该优化问题.文中还建立MapReduce作业的Map任务和Reduce任务性能与内存的关系模型,求解最佳性能的内存需求,从而获得Map任务和Reduce任务的年长代内存大小;使用聚类算法预测JVM晋升对象阈值,优化JVM配置,减少了JVM的垃圾回收暂停时间.实验结果表明,文中提出的方法能准确预测作业的内存需求,显著提升作业运行性能.
[Abstract]:There is a great correlation between MapReduce job performance and memory configuration. Aiming at the difficulty of accurately predicting job memory, a generation memory prediction method is proposed according to the characteristics of generation memory management of Java Virtual Machine (JVM). First, the relationship between the young generation and the average garbage collection time is modeled by using the regression model. The problem of finding the reasonable memory size of the younger generation is transformed into a constrained nonlinear optimization problem, and a search algorithm is designed to solve the optimization problem. In this paper, the relationship model of Map task and Reduce task performance and memory of MapReduce job is established to solve the memory requirement of the best performance, so as to obtain the older generation memory size of Map task and Reduce task. The clustering algorithm is used to predict the threshold of JVM promotion object, optimize the JVM configuration, and reduce the garbage collection pause time of JVM. The experimental results show that the proposed method can accurately predict the memory requirements of jobs and significantly improve the performance of jobs.
【作者单位】：四川大学网络空间安全研究院;
【基金】：国家科技支撑计划项目(2012BAH18B05) 国家自然科学基金资助项目(61272447)~~
【分类号】：TP311.13

【相似文献】