一种MapReduce架构下基于遗传算法的K-Medoids聚类

发布时间：2018-06-27 20:29

本文选题：海量数据 + K-Medoids　；参考：《计算机科学》2017年03期

【摘要】：由互联网时代快速发展而产生的海量数据给传统聚类方法带来了巨大挑战,如何改进聚类算法从而获取有效信息成为当前的研究热点。K-Medoids是一种常见的基于划分的聚类算法,其优点是可以有效处理孤立、噪声点,但面临着初始中心敏感、容易陷入局部最优值、处理大数据时的CPU和内存瓶颈等问题。为解决上述问题,提出了一种MapReduce架构下基于遗传算法的K-Medoids聚类。利用遗传算法的种群进化特点改进K-Medoids算法的初始中心敏感的问题,在此基础上,利用MapReduce并行遗传K-Medoids算法提高算法效率。通过带标签的数据集进行实验的结果表明,运行在Hadoop集群上的基于MapReduce和遗传算法的K-Medoids算法能有效提高聚类的质量和效率。
[Abstract]:The huge amount of data generated by the rapid development of the Internet era has brought great challenges to the traditional clustering methods. How to improve the clustering algorithm to obtain effective information has become the current research hotspot. K-Medoids is a common partition-based clustering algorithm. Its advantage is that it can deal with isolated and noise points effectively, but it is faced with some problems such as initial center sensitivity, easy to fall into local optimal value, CPU and memory bottleneck when processing big data. In order to solve the above problems, a genetic algorithm based K-Medoids clustering based on MapReduce is proposed. The problem of initial center sensitivity of K-Medoids algorithm is improved by using the characteristics of population evolution of genetic algorithm. On this basis, MapReduce parallel genetic K-Medoids algorithm is used to improve the efficiency of the algorithm. The experimental results of tagged data sets show that the K-Medoids algorithm based on MapReduce and genetic algorithm running on Hadoop cluster can effectively improve the quality and efficiency of clustering.
【作者单位】：天津大学计算机科学与技术学院;天津市认知计算与应用重点实验室;
【基金】：国家自然科学基金项目(61170177) 国家重点基础研究发展计划项目(2013CB32930X)资助
【分类号】：TP311.13;TP18

【参考文献】