分布式KNN算法在微信公众号分类中的应用

发布时间：2018-08-23 21:16

【摘要】：针对微信公众号数据量大幅增长与从事微信活动的人们对其有效信息获取效率低下的问题,提出对微信公众号信息进行梳理并快速并行化分类以及打标签的方法。首先,该方法在介绍微信公众号实际应用的前提下,以经典K最近邻(KNN)分类算法为基础,实践并分析了单机KNN算法在效率上的不足;然后,采用Hadoop平台实现了基于MapReduce模型的KNN算法,对比了单机与分布式的效率以及对K值的调优,实验中的样本训练集通过人为指定,文本相似度的判别分为分词、特征词提取、权重计算、测试向量与训练向量夹角计算等步骤。在24个类别基础上,通过对1 000万条公众号数据分类实验,为每个公众号打上了单标签或多标签,优化后的分类准确率达到82%,其中与生活相关的公众号数量占比达70%以上。研究表明使用分类后的结果,信息针对特定人群传播,传播的转化率有所提升;分布式KNN算法在微信公众号数据处理方面比单机算法具有更高的效率和鲁棒性。
[Abstract]:In view of the problem that the amount of data of public number of WeChat increases greatly and the people engaged in WeChat activities have low efficiency in obtaining effective information, a method of sorting and tagging the information of public number of WeChat is put forward. First of all, on the basis of introducing the practical application of WeChat Public number, based on the classical K-nearest neighbor (KNN) classification algorithm, the paper puts into practice and analyzes the efficiency deficiency of the single-machine KNN algorithm. The KNN algorithm based on MapReduce model is implemented by using Hadoop platform. The efficiency of single machine and distributed system and the optimization of K value are compared. The sample training set is artificially assigned, and the text similarity can be divided into word segmentation and feature word extraction. Weight calculation, test vector and training vector angle calculation and other steps. On the basis of 24 categories, 10 million public number data were classified with single label or multiple label. The accuracy of the optimized classification reached 82%, in which the number of public numbers related to life accounted for more than 70% of Prida. The results show that the information is transmitted to a specific population and the conversion rate is improved. The distributed KNN algorithm is more efficient and robust than the single algorithm in the public number data processing of WeChat.
【作者单位】：西南石油大学计算机科学学院;
【基金】：国家安全生产总局项目(sichuan-0008-2016AQ,sichuan-0009-2016AQ)
【分类号】：TP391.1

【参考文献】