微博垃圾评论识别方法研究

发布时间：2018-03-21 11:46

本文选题：微博垃圾评论　切入点：协同训练　出处：《广西师范大学》2017年硕士论文　论文类型：学位论文

【摘要】：垃圾评论是指用户发布的与博文无关的,或没有意义的,或蓄意发表的评论信息。早期常采用人工识别方法来识别,主要有基于验证码、审核机制两种;中期采用自动识别方法来识别,主要有基于关键词、基于链接数量和基于相关度阈值的方法;近期先采用基于规则方法过滤掉超链接、特殊字符等明显的显式垃圾评论,然后采用基于主题特征的方法结合分类器进行微博垃圾评论识别。目前常采用的微博数据获取方法主要有网络爬虫和微博开放平台API两种,前者速度慢,处理本文所需实验数据需要花费大量的时间,而后者的访问次数受到微博平台服务器的限制,两种方法获取实验数据都不是很理想。所以,本文提出了一种基于cookie与正则表达式的方法获取实验所需的数据,包括微博原文、微博作者信息和微博评论。本文设计采用以上两种常用方法和本文提出方法获取经过微博认证的用户名为王宝强发表的主题为离婚微博的评论数据,实验结果表明,本方法相较于两种常用方法,不仅操作相对简单,而且数据获取速度较快。微博及其评论的字符受到限制,最多只有140字,内容相对短小,微博的主题特征并不是特别明显,对微博中垃圾评论进行识别不能只考虑评论和微博之间的相关程度,因为单一的因素考虑会增加垃圾评论的误判率。因此,本文尝试使用协同训练方法来增强分类器性能,提出一种基于Co-Training协同训练的垃圾评论识别方法。对于微博原文和微博作者信息,本文进行预处理后得到的相关信息词组,与微博特有情感词汇以及大连理工信息检索实验室的情感词汇本体中情感强度大于5的情感词构成特征词汇库。对于微博评论,本文通过定义的基于规则识别方法过滤出显式垃圾评论,对于剩下的相关评论进行预处理后,一方面,得到相关评论词组,和构造的特征词汇库通过同义词词林相似度计算方法计算出结果,送入AdaBoost分类器,另一方面,进行特征提取,得到评论特征作为特征向量来训练SVM分类器。最后将两分类器通过基于微博垃圾评论的Co-Training协同训练算法进行协同训练,用训练好的模型来判断评论是否为垃圾评论。本方法在提高分类精度的同时,节省了大量的样本标注工作,通过实验将本文方法和其他两种典型的方法进行比较分析,结果表明本文提出的方法具备良好的可行性和有效性。
[Abstract]:Spam comment refers to the comment information issued by users which has nothing to do with blog posts or is meaningless or intentionally published. In the early stage manual identification is often used to identify the spam comments. There are two kinds of comment information based on verification code and verification mechanism. In the middle stage, automatic recognition method is adopted, which is mainly based on keyword, link number and relevance threshold. In recent years, rules based method is used to filter out hyperlinks, special characters and other obvious explicit spam comments. Then we use theme-based feature based method combined with classifier to recognize Weibo's garbage comment. At present, there are two kinds of common data acquisition methods, namely web crawler and Weibo open platform API, which are slow in speed. It takes a lot of time to process the experimental data required in this paper, and the number of access to the latter is limited by Weibo platform server. Neither method is ideal for obtaining experimental data. In this paper, a method based on cookie and regular expression is proposed to obtain the experimental data, including Weibo's original text. Weibo author's Information and Weibo comments. This paper designs to use the above two common methods and the method proposed in this paper to obtain the comment data on the subject of "divorcing Weibo" published by Wang Baoqiang, whose user name has been authenticated by Weibo. The experimental results show that, Compared with the two common methods, this method is not only relatively simple to operate, but also faster to obtain data. Weibo and his comments are limited in characters, at most 140 words, and the content is relatively short. The thematic features of Weibo are not particularly obvious. The identification of spam comment in Weibo can not only consider the correlation between comment and Weibo, because a single factor will increase the misjudgment rate of garbage comment. Therefore, this paper tries to use the cooperative training method to enhance the performance of classifier. A method of garbage comment recognition based on Co-Training cooperative training is proposed. With Weibo and the emotion vocabulary of Dalian University of Science and Technology Information Retrieval Laboratory, emotion words with more than 5 emotional intensity constitute the characteristic vocabulary database. In this paper, explicit spam comments are filtered out by the defined rule-based recognition method. After preprocessing the remaining related comments, on the one hand, the relevant comment phrases are obtained. And the constructed feature vocabulary database calculates the result by calculating the similarity degree of synonym forest, and sends it into the AdaBoost classifier, on the other hand, carries on the feature extraction, Finally, the two classifiers are trained by the Co-Training co-training algorithm based on Weibo spam comment, which is used as the feature vector to train the SVM classifier. This method not only improves the classification accuracy, but also saves a lot of sample labeling work. Through experiments, the method is compared with other two typical methods. The results show that the proposed method is feasible and effective.
【学位授予单位】：广西师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092;TP391.1

【相似文献】