基于分布式计算的数据挖掘算法研究与实现

发布时间：2018-12-18 01:39

【摘要】：随着互联网访问便捷性的提高,互联网的线上活动已经成为一个越来越受欢迎的新兴领域。互联网的快速发展,扩大了互联网的应用领域。由此,互联网行业也产生了大量的用户数据。传统的单机计算方式,已经逐渐难以满足互联网行业实际业务情景下的计算需求和计算速度要求。而基于分布式计算的数据挖掘算法研究,有助于在互联网数据量日益增多的今天发挥其在计算能力和处理速度的优势。这就要求人们转换传统单机计算数据挖掘算法的设计思想,实现分布式计算的数据挖掘算法。为了实现这一要求,本文提出基于分布式计算的数据挖掘研究方法。本方法基于单机数据挖掘算法原理,对目前最为广泛使用的分类算法——朴素贝叶斯分类算法、SVM分类算法,关联规则——FP-Growth和聚类算法——Canopy算法、k-Means聚类算法来进行基于分布式计算的数据挖掘算法研究和实现,并将基于分布式朴素贝叶斯算法和FP-Growth关联规则的文本分类以及基于分布式环境的改进k-Means算法的聚类分析应用在微博热点博文分析系统中。本文的主要工作如下:1.研究数据挖掘算法的基本理论和分布式计算的基本设计思想,提出了本文的重点研究内容——基于分布式计算的数据挖掘算法,即分布式环境中的分类算法——朴素贝叶斯算法、SVM算法,关联规则——FP-Growth 以及聚类算法 k-Means、Canopy、改进 k-Means 聚类算法;2.基于上一步提出的研究内容,本文对分布式环境中的数据挖掘算法进行研究。本方法,首先,在充分研究数据挖掘算法的基础上,结合分布式环境Hadoop中的MapReduce编程模型的特点来实现基于分布式环境的朴素贝叶斯分类算法、SVM分类算法、关联规则FP-Growth、Canopy聚类算法、k-Means聚类算法以及改进的k-Means聚类算法。基于对分布式计算数据挖掘算法的实现,针对不同分布式数据挖掘算法对经典数据集进行实验对比,分析基于分布式计算的数据挖掘算法的处理效率等方面指标;3.基于上述分布式环境中的数据挖掘方法的实验结果和分析,本文设计并实现了微博热点博文分析系统。实验表明,本方法能够满足微博热点博文分析系统各模块的基本功能,并验证了分布式数据挖掘算法相对于单机计算的性能优势。微博热点博文分析系统首先结合分布式环境中的朴素贝叶斯算法、分类规则算法来对微博博文数据进行主题分类,然后结合本文提出的分布式环境中数据挖掘算法的改进k-Means算法来对基于主题的微博数据进行微博热点博文分析,最后根据博文分析结果对各项评价指标进行分析。
[Abstract]:With the improvement of Internet access convenience, the online activities of the Internet have become an increasingly popular emerging field. With the rapid development of the Internet, the application of the Internet has been expanded. As a result, the Internet industry has also produced a large number of user data. The traditional single computer computing method has been gradually difficult to meet the actual business situation of the Internet industry computing requirements and computing speed requirements. The research of data mining algorithm based on distributed computing is helpful to give full play to its advantage in computing power and processing speed in today's Internet data volume increasing day by day. This requires people to change the design idea of traditional single-machine computing data mining algorithm and realize the distributed computing data mining algorithm. In order to meet this requirement, this paper proposes a research method of data mining based on distributed computing. This method is based on the principle of single machine data mining algorithm. At present, the most widely used classification algorithms are naive Bayes classification algorithm, SVM classification algorithm, association rule FP-Growth and clustering algorithm Canopy algorithm. K-Means clustering algorithm is used to research and implement the data mining algorithm based on distributed computing. The text classification based on distributed naive Bayes algorithm and FP-Growth association rules and the clustering analysis of improved k-Means algorithm based on distributed environment are applied to Weibo hot spot blog analysis system. The main work of this paper is as follows: 1. The basic theory of data mining algorithm and the basic design idea of distributed computing are studied. That is, naive Bayesian algorithm, SVM algorithm, association rule FP-Growth and k-Means-Canopyalgorithm, which are the classification algorithms in distributed environment, improve the k-Means clustering algorithm. 2. Based on the previous research content, this paper studies the data mining algorithm in distributed environment. In this method, first of all, based on the research of data mining algorithm, combining the characteristics of MapReduce programming model in distributed environment Hadoop, the naive Bayes classification algorithm, SVM classification algorithm and association rule FP-Growth, are implemented based on distributed environment. Canopy clustering algorithm, k-Means clustering algorithm and improved k-Means clustering algorithm. Based on the implementation of distributed computing data mining algorithm, this paper compares the classical data sets with different distributed data mining algorithms, and analyzes the processing efficiency of data mining algorithms based on distributed computing. 3. Based on the experimental results and analysis of the data mining methods in the distributed environment mentioned above, this paper designs and implements Weibo hot spot blog analysis system. Experiments show that this method can meet the basic functions of Weibo hot spot blog analysis system and verify the performance of distributed data mining algorithm compared with single computer. Weibo Hot spot blog Analysis system first combines naive Bayes algorithm and classification rule algorithm in distributed environment to classify the topic of Weibo blog data. Then combine the improved k-Means algorithm of data mining algorithm in distributed environment to analyze the Weibo data based on topic, then analyze the hot spot blog on the basis of the analysis result of blog. Finally, the evaluation index is analyzed according to the result of the analysis.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】