基于VSM-BTM主题模型的微博热点话题发现研究

发布时间：2018-04-12 07:42

本文选题：微博 + 话题检测　；参考：《西南大学》2017年硕士论文

【摘要】：随着互联网的飞速发展,微博作为一种社交媒体已经获得了社会各界的广泛关注。但是如何从海量、不规则的微博数据中高效地提取出有效的信息来进行话题发现,仍然是目前亟待解决的问题。因此,使用主题模型挖掘微博数据的方法得以产生。目前,学者们已经对于主题模型进行了大量的研究,但现有的方法技术仍然存在一些不足,主要体现在:一是计算的复杂度太高,对于大数据级别的微博数据计算的效率不高;二是使用一些主题模型(比如传统的LDA模型)对微博这种短文把数据进行聚类后的准确度不高等。基于此,本文提出了一种融入改进的VSM模型和BTM主题模型和改进的适合微博数据的K-Means聚类方法的新浪微博数据挖掘方法,在保证计算微博数据效率的同时,提高微博数据挖掘的准确度。本文对VSM-BTM主题模型的微博数据挖掘方法进行了研究,研究内容主要分为微博数据的预处理、VSM-BTM建模、适合微博的聚类方法三个部分。其中,微博数据的预处理包括分词、去停用词、删除噪音数据等过程,并将预处理的结果以txt格式的文本保存,作为下一步主题建模的输入。在VSM-BTM建模过程中,首先使用现有的BTM主题模型进行建模,对数据预处理的结果进行不断地迭代,得到“文档-主题”矩阵和“主题-词语”矩阵,同时,利用BTM主题模型生成的词库表和微博数据转码结果,提出了一种使用JS距离和余弦距离相结合计算微博数据之间的相似度的方法。适合微博的聚类方法是使用适合微博数据的改进K-Means聚类方法对建模结果进行聚类分析,主要是通过利用现有微博数据选取合适的初始簇和计算距离的方法对传统的K-Means聚类算法进行了改进。最后采用准确率、召回率和F1值对实验结果进行分析评价。使用VSM-BTM主题模型进行建模的方法避免了微博数据稀疏性的缺陷,且不需要使用外部信息对微博数据进行扩充,降低了对文本以外信息的依赖性。通过实验,本文对单纯的LDA主题模型、单纯的BTM主题模型和本文提出的VSM-BTM主题模型的微博热点话题发现效果进行对比分析,以3个主题模型的准确率、召回率和F1值为对比分析的依据,发现本文提出的VSM-BTM主题模型在各个评价因素中都优于单纯的LDA主题模型和单纯的BTM主题模型的微博热点话题发现效果,从而证明了本文使用的主题模型对微博数据进行建模和聚类方法的有效性,在不增加计算复杂度的前提下,准确度优于现有的其他两种微博数据挖掘方法。
[Abstract]:With the rapid development of Internet, Weibo, as a kind of social media, has received wide attention from all walks of life.However, how to efficiently extract effective information from massive and irregular Weibo data for topic discovery is still an urgent problem.Therefore, the method of mining Weibo data using topic model can be produced.At present, scholars have done a lot of research on thematic models, but the existing methods and techniques still have some shortcomings, mainly reflected in: first, the complexity of the calculation is too high, the efficiency of the big data level Weibo data calculation is not high;Secondly, some thematic models (such as the traditional LDA model) are used to cluster the data of Weibo.Based on this, this paper proposes an improved VSM model and BTM topic model and an improved K-Means clustering method suitable for Weibo data mining, which ensures the efficiency of data calculation.Improve the accuracy of Weibo data mining.In this paper, the Weibo data mining method of VSM-BTM topic model is studied. The research content is divided into three parts: the pretreatment of Weibo data and the modeling of VSM-BTM, and the clustering method suitable for Weibo.Among them, the pretreatment of Weibo data includes participle, deactivation word, noise data and so on. The result of the preprocessing is saved as the text of txt format as the input of the next topic modeling.In the process of VSM-BTM modeling, we first use the existing BTM topic model to model, iterate over the results of data preprocessing, get the "document-topic" matrix and "subject-word" matrix, and at the same time,Based on the lexical table generated by the BTM subject model and the result of Weibo data transcoding, a method of calculating the similarity between Weibo data using JS distance and cosine distance is proposed.The clustering method suitable for Weibo is to use the improved K-Means clustering method, which is suitable for Weibo data, to analyze the modeling results.The traditional K-Means clustering algorithm is improved by using the existing Weibo data to select suitable initial clusters and calculate the distance.Finally, the accuracy rate, recall rate and F1 value are used to analyze and evaluate the experimental results.The method of modeling with VSM-BTM topic model avoids the limitation of Weibo's data sparsity, and does not need to use external information to expand Weibo data, thus reducing the dependence on information other than text.Through the experiments, this paper makes a comparative analysis of the effect of Weibo hot topic discovery between the pure LDA theme model, the simple BTM theme model and the VSM-BTM theme model proposed in this paper. The accuracy of the three thematic models is compared and analyzed.Recall rate and F1 value are the basis of comparative analysis. It is found that the VSM-BTM theme model proposed in this paper is better than Weibo hot topic discovery effect of LDA theme model and BTM theme model in all evaluation factors.It is proved that the thematic model used in this paper is effective in modeling and clustering Weibo data, and the accuracy is superior to that of the other two Weibo data mining methods without increasing computational complexity.
【学位授予单位】：西南大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.1;TP393.092

【参考文献】