基于资源签名与遗传算法的Hadoop参数自动调优系统

发布时间：2018-01-17 16:31

本文关键词：基于资源签名与遗传算法的Hadoop参数自动调优系统　出处：《计算机应用研究》2017年11期 　论文类型：期刊论文

【摘要】：在Hadoop集群的优化配置中,配置参数存在种类繁多、含义复杂、相互关联影响的特性,导致难以实现快速准确寻优。针对以上问题,构建了Hadoop集群自动调优系统,其中在系统中设计了资源获取器与参数配置库,分别用于获取各作业的资源消耗与存储分发配置方案。该系统利用MapReduce作业的小规模数据集资源签名将任务分类,在遗传算法框架中通过任务的测试评估对配置方案进行自动迭代优化。实验结果表明,调优后集群的任务完成时间明显缩减,集群的资源利用率有了明显提升。
[Abstract]:In the optimal configuration of Hadoop cluster, there are many kinds of configuration parameters, complex meanings and interrelated effects, which makes it difficult to achieve fast and accurate optimization. The automatic tuning system of Hadoop cluster is constructed, in which resource acquirer and parameter configuration library are designed. It is used to obtain the resource consumption and storage distribution configuration scheme of each job respectively. The system classifies the task by using the small scale data set resource signature of the MapReduce job. In the framework of genetic algorithm (GA), the task testing and evaluation are used to optimize the configuration scheme automatically. The experimental results show that the task completion time of the optimized cluster is significantly reduced and the resource utilization of the cluster has been significantly improved.
【作者单位】：中国科学院沈阳计算技术研究所;中国科学院大学;
【基金】：国家科技重大专项资助项目(2012ZX01029001-002,2014ZX04009031)
【分类号】：TP18;TP302.1
【正文快照】： Hadoop[1]是Apache软件基金会(Apache Software Founda-tion)旗下的一个开源分布式计算平台,具有高易用性、高容错性和高可扩展性等优点,可以搭建在廉价的普通机器上构成集群。随着Hadoop分布式平台在大规模数据存储与处理中越来越普及,集群的运算与存储压力也越来越大,在有限

【相似文献】