基于微博的热点话题提取

发布时间：2018-09-14 18:09

【摘要】：随着互联网的高速发展,微博作为一种信息平台,以其庞大的用户群、特有的用户关系结构显示出巨大的影响力,在人们的社交生活中扮演着越来越重要的角色,已经成为信息传播的新势力。目前,仅新浪微博一天的微博发布量就达到了几千万甚至上亿条,这种量级的数据很难通过人工处理的方法及时提取出隐藏在海量数据背后的热点话题。因此,使用计算机自动处理微博信息,及时从海量信息中挖掘出热点话题,对于了解最新的舆论热点、掌握舆论动向有着重大意义。传统的TF-IDF话题提取方法,由于其特征维度较高、数据较稀疏,无法从语义层面解释词与词之间的关系。概率主题模型LDA(Latent Dirichlet Allocation)认为每个文档可以包含多个主题,不同主题下对应的词的生成概率不同。相比于其它的文本模型,LDA更符合实际应用中的情况,对文本有着更好的描述能力。本文针对微博话题的挖掘和提取进行了研究,具体的工作包括： 1.通过研究各种文本建模方法,选取了LDA作为最终的模型。通过使用吉布斯抽样方法求解LDA模型,得到了微博文本的主题分布向量。使用主题分布向量作为微博的文本特征有效地降低了数据的维度,为后续的聚类算法提供了维度低、区‘分度高的数据。 2.对Single-Pass聚类算法进行了改进,在保证聚类效果的同时提升了聚类的时间效率。 3.研究了文本类的话题词提取算法,提出了基于词共现模型的相似度度量方法,使用相似度矩阵进行层次聚类分析,选取最大的类作为最能代表微博文本类内容的话题词组。 4.完成了热点话题提取系统,有机组合了网络爬虫、数据库模块、分词模块、聚类模块、话颗词提取模声,实现了微博热点话题的自动提取。
[Abstract]:With the rapid development of the Internet, Weibo, as an information platform, with its huge user group and unique user relationship structure, has shown great influence and played a more and more important role in people's social life. Has become a new force in the dissemination of information. At present, Sina Weibo alone released tens of millions or even hundreds of millions of Weibo in one day. It is very difficult to extract hot topics hidden behind massive data by manual processing. Therefore, it is of great significance to use the computer to process Weibo information automatically and to dig out hot topics from mass information in time, which is of great significance to understand the latest hot spots of public opinion and grasp the trend of public opinion. Because of its high feature dimension and sparse data, the traditional TF-IDF topic extraction method can not explain the relationship between words and words from the semantic level. The probabilistic topic model (LDA (Latent Dirichlet Allocation) holds that each document can contain more than one topic, and the generation probability of the corresponding words under different topics is different. Compared with other text models, LDA is more suitable for practical applications and has better description ability. This paper studies the topic mining and extraction of Weibo, the specific work includes: 1. By studying various text modeling methods, LDA is selected as the final model. By using Gibbs sampling method to solve the LDA model, the theme distribution vector of Weibo text is obtained. Using topic distribution vector as the text feature of Weibo can effectively reduce the dimension of data, and provide the data with low dimension and high 'grade' for the subsequent clustering algorithm. 2. The Single-Pass clustering algorithm is improved, which not only ensures the clustering effect, but also improves the time efficiency of clustering. This paper studies the topic word extraction algorithm of text class and proposes a similarity measure method based on word co-occurrence model. The similarity matrix is used for hierarchical clustering analysis. Select the largest class as the most representative of Weibo text content topic phrase. 4. A hot topic extraction system is completed, which combines web crawler, database module, word segmentation module, clustering module, word extraction module, and realizes Weibo hot topic extraction automatically.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP391.1;TP393.092

【参考文献】