基于Hadoop平台的LDA短文本分类算法研究

发布时间：2019-03-04 13:13

【摘要】：近年来,随着即时通讯、微博等网络应用的发展,大量长度较短的文本信息应运而生。这些数据不仅增长速度快,且数量庞大。如何合理利用海量文本数据,从中提取有价值的信息,便成为了当下的研究热点。基于短文本的研究目前在网络舆情分析、热点话题发现、社交网络、购物平台推荐及信息安全等各个领域都受到广泛应用。短文本信息具有内容长度短、特征稀疏、噪点多等特性,以至于传统的文本分类方法效果非常不理想。本文在前人的研究基础上,提出了一种基于共现关系LDA主题的短文本分类方法。利用潜在狄利克雷主题模型(Latent Dirichlet Allocation,LDA)对短文本进行处理,得到"主题一词"分布;再提取同时出现在多个主题中的词语,建立共现词集;然后通过计算共现词集中各个词语与各个主题的相关度,将与两个以上主题具有近似相关度的词语做进一步筛选,建立混淆词集。在进行文本分类时,对于混淆词集中的词语,通过降低其权重来减少对分类结果的影响。为了提高本文方法的运行效率,将本文方法与Hadoop平台相结合,利用Hadoop分布式系统在处理海量数据的优势,来优化本文分类方法的分类效率。文本实验使用两种语料库:新闻标题语料库和微博语料库。实证过程制定两种实验方案:先使用样本较小的新闻标题语料库进行算法可行性验证,通过与其他方法进行对比,验证本文方法在分类效果方面的优势;随后利用大样本的微博语料库在Hadoop平台下测试本文方法是否在分类效率方面具有显著提高。最后通过实验结果分析得出,文本提出的基于共现关系的LDA短文本分类方法以及将此分类方法与Hadoop平台结合的分类效果效率均实现预期目标。
[Abstract]:In recent years, with the development of instant messaging, Weibo and other network applications, a large number of short-length text information emerged as the times require. These data not only grow fast, but also a large number. How to make rational use of massive text data and extract valuable information from it has become a hot research topic. Short text-based research is widely used in network public opinion analysis, hot topic discovery, social network, shopping platform recommendation and information security and so on. The short text information has the characteristics of short content length, sparse features and many noise points, so that the traditional text classification method is not ideal. On the basis of previous research, this paper proposes a short text classification method based on co-occurrence relation LDA topic. The latent Dirichlet theme model (Latent Dirichlet Allocation,LDA) is used to process the short text to obtain the distribution of the word "theme", and then to extract the words that appear in multiple topics at the same time, and to set up a set of co-existing words. Then by calculating the correlation degree between each word and each topic in the co-occurrence word set, the words with approximate correlation degree with more than two topics are further screened, and the confused word set is established. When text classification is carried out, the influence on the classification result is reduced by reducing the weight of the confused words in the set of words. In order to improve the efficiency of this method, this paper combines this method with Hadoop platform, and makes use of the advantage of Hadoop distributed system in processing massive data to optimize the classification efficiency of this classification method. The text experiment uses two kinds of corpus: news title corpus and Weibo corpus. In the empirical process, two kinds of experimental schemes are formulated: firstly, the feasibility of the algorithm is verified by using the corpus of news headlines with smaller samples, and the advantages of this method in classification effect are verified by comparing with other methods; Then a large sample of Weibo corpus is used to test whether the proposed method has a significant improvement in classification efficiency under the Hadoop platform. Finally, through the analysis of experimental results, it is concluded that the proposed LDA short text classification method based on co-occurrence relation and the efficiency of combining this classification method with Hadoop platform can achieve the desired goal.
【学位授予单位】：天津财经大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.1

【相似文献】