基于LDA模型的微博话题与事件检测

发布时间：2018-04-03 10:08

本文选题：微博　切入点：话题检测　出处：《哈尔滨工业大学》2014年硕士论文

【摘要】：当前,我们处在一个瞬息万变的互联网时代。近几年来,以SNS、微博、微信等为代表的新型网络社交工具迅速崛起,各自拥有了数量巨大的用户群体。微博凭借着实时性强,内容简练(140字以内)和发布方式多样等优点,已经成为网上信息发布和传播的主要平台之一。微博在短时期内就能够聚集大量的文本数据,如何在这些杂乱、无序的微博文本数据中快速地提取出精炼的、有价值的话题,是一项艰巨的任务,需要对现有的话题检测技术进行发展和提高。本文提出了一种基于LDA-SP(Latent Dirichlet Allocation-Single Pass)的微博话题检测算法。首先分析了话题检测的基本流程,阐述了各个环节使用技术的基本原理和实现细节。针对在传统话题检测中,以向量空间模型作为文本模型表示存在着维度过高、语义表现缺失等缺点,本文改进了传统方法,采用潜在狄利克雷分配模型对微博文本建模,采用Single-Pass算法作为微博话题检测中聚类的实现方法,将两者结合使用。对照实验的结果表明,本文提出的算法在解决了预设话题数的缺点的同时,还保证了话题检测的精度。本文提出了一种微博事件的同一性计算方法。该方法用来区分微博数据集中具有相似内容的不同事件。由于这种“难分”问题的存在,主题模型无法对相似内容的不同事件进行分辨。本文首先考虑两条语义上相似的微博,然后计算它们在时间、地点等事件特征上的同一性评分,从而推断出它们是否表述同一个事件。将此方法与Single-Pass算法相结合,最后引入微博数据集进行实验。实验表明,相比之前的算法,改进相似度后的算法在“难分”问题上具有更好的事件检测效果。
[Abstract]:At present, we are in a rapidly changing era of the Internet.In recent years, with SNS, Weibo, WeChat as the representative of the new social networking tools, each has a large number of users.Weibo has become one of the main platforms for information dissemination and dissemination on the Internet because of its advantages of real time, concise content within 140 words) and variety of publishing methods.Weibo was able to gather a large amount of text data in a short period of time. It is a difficult task to quickly extract refined and valuable topics from these messy and disordered Weibo text data.It is necessary to develop and improve the existing topic detection technology.This paper presents a topic detection algorithm for Weibo based on LDA-SP(Latent Dirichlet Allocation-Single Pass.Firstly, the basic flow of topic detection is analyzed, and the basic principle and implementation details of each link are expounded.In traditional topic detection, vector space model (VSM) as a text model has some disadvantages, such as too high dimension, lack of semantic representation, etc. In this paper, we improve the traditional method and use latent Delikley assignment model to model Weibo text.Single-Pass algorithm is used as a clustering method in Weibo topic detection, and the two methods are combined.The experimental results show that the proposed algorithm not only solves the shortcomings of the preset number of topics, but also ensures the accuracy of topic detection.This paper presents a method for calculating the identity of Weibo event.This method is used to distinguish different events with similar contents in Weibo dataset.Because of this problem, the topic model can not distinguish different events with similar content.In this paper, we first consider two syntactically similar Weibo, and then calculate their identity scores in time, place and other event features, and then infer whether they represent the same event or not.This method is combined with Single-Pass algorithm, and then Weibo data set is introduced to experiment.Experiments show that the improved similarity algorithm has better event detection performance than the previous algorithm.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP391.1;TP393.092

【相似文献】