基于Hadoop的关联规则挖掘算法研究——以Apriori算法为例

发布时间：2018-12-29 07:39

【摘要】：为了解决传统关联规则挖掘算法在挖掘效率、算法扩展性等方面无法适应大数据挖掘需求的问题,以经典的关联规则挖掘算法—Apriori算法为例,首先基于Hadoop平台和MapReduce编程模型,实现算法的并行化。在此基础上,基于事务缩减的思想对算法进行优化,进一步提高算法的挖掘效率。搭建Hadoop集群环境,对算法的挖掘结果和挖掘效率进行实验。通过并行挖掘结果验证、串行版与并行版效率对比、挖掘时间与节点数目的变化关系、挖掘时间与数据量的变化关系4组实验,结果表明:文中实现的Apriori算法不仅能够准确挖掘频繁项集,而且比传统串行算法具有更高的挖掘性能和可扩展性。该算法能够更好地适应大数据集的挖掘要求,能够实现从大规模数据集中高效挖掘频繁项集和关联规则。
[Abstract]:In order to solve the problem that the traditional association rules mining algorithm can not adapt to big data mining demand in mining efficiency and algorithm expansibility, taking the classical association rule mining algorithm-Apriori algorithm as an example, firstly, it is based on Hadoop platform and MapReduce programming model. The parallel algorithm is realized. On this basis, the algorithm is optimized based on the idea of transaction reduction to further improve the efficiency of the algorithm. The Hadoop cluster environment is built and the mining results and efficiency of the algorithm are tested. Through the verification of parallel mining results, the efficiency of serial version and parallel version is compared, the relationship between mining time and the number of nodes, and the relationship between time and data are explored in four groups of experiments. The results show that the proposed Apriori algorithm not only can mine the frequent itemsets accurately, but also has higher mining performance and scalability than the traditional serial algorithms. The algorithm can better meet the mining requirements of big data sets and can efficiently mine frequent itemsets and association rules from large-scale data sets.
【作者单位】：南京大学信息管理学院;
【基金】：国家自科基金面上项目(71473114)
【分类号】：TP311.13

【相似文献】