微博数据挖掘可视化系统的设计与实现

发布时间：2019-05-23 14:51

【摘要】：随着移动通讯网络环境的不断完善以及智能手机的进一步普及,我国互联网已全面进入Web2.0时代。作为Web2.0的典型代表,微博拥有大量的活跃用户,内容覆盖领域广泛,社会影响力巨大。微博已经成为了人们获取信息、分享观点的重要渠道,海量数据背后蕴藏着巨大的学术研究价值。因此,本文以微博为研究对象,围绕微博数据的采集、挖掘、情感分析和可视化进行研究,设计并实现基于微博的数据挖掘可视化系统。本文的主要工作包括:(1)在数据采集方面,设计并实现了微博爬虫系统。该系统采用模拟登录解决身份认证问题,参考广度优先搜索的思想,利用热门微博监测模块实现高质量用户的自动发现,并结合网络爬虫、BeautifulSoup、正则表达式、多线程并发和数据库等技术实现多种用户信息和微博信息的采集。该爬虫系统解决了信息采集不全面、请求微博服务器过于频繁的问题,实现了微博数据全面高效的获取。(2)在数据挖掘方面,设计并实现了微博数据挖掘的用户分析模块和微博分析模块,提供了微博分析的基本功能,并基于机器学习算法重点研究了微博文本的情感分析,设计并实现了分类器的训练实验。本文分别采用“单词”、“双词”和“单词双词结合”三种特征抽取模型,利用卡方统计算法进行特征选择,并采用朴素贝叶斯、逻辑回归和支持向量机等六种分类算法进行对比实验。通过反复实验比较,获得了最优的分类模型。该模型无论对微博文本还是更短小的评论文本均得到了较好的分类效果。(3)在数据可视化方面,本文采用柱状图、折线图、地图、标签云、饼图、仪表盘等多种可视化图表展示数据分析结果,并通过浏览器呈现。本系统采用B/S结构,前端采用浏览器展示分析结果,后台由微博爬虫、MySQL关系型数据库、数据挖掘模块组合实现数据的采集、处理和分析功能。最终,实现微博的数据挖掘与可视化分析。本文的主要贡献和创新点包括:(1)设计并实现了包括微博数据采集、数据挖掘和数据可视化的微博分析系统。搭建的系统实现了用户分析和微博分析等功能,为后续研究提供了基础平台。(2)系统实现了微博文本的情感分析功能,本文采用机器学习算法训练情感分析模型,该模型的正确率达到85%,AUC值达到0.94。系统可以直接调用该分类器实现微博文本的情感分析。
[Abstract]:With the continuous improvement of mobile communication network environment and the further popularization of smart phones, China's Internet has entered the Web2.0 era in an all-round way. As a typical representative of Web2.0, Weibo has a large number of active users, a wide range of content coverage, great social influence. Weibo has become an important channel for people to obtain information and share views, and there is a huge academic research value behind the massive data. Therefore, this paper takes Weibo as the research object, focuses on Weibo data collection, mining, emotional analysis and visualization, and designs and implements a data mining visualization system based on Weibo. The main work of this paper is as follows: (1) in the aspect of data acquisition, the Weibo crawler system is designed and implemented. The system adopts simulated login to solve the identity authentication problem, refers to the idea of breadth first search, uses the popular Weibo monitoring module to realize the automatic discovery of high quality users, and combines the network crawler and BeautifulSoup, regular expression. Multi-thread concurrency and database technology realize the collection of various user information and Weibo information. The crawler system solves the problem that the information collection is not comprehensive and the Weibo server is too frequent, and realizes the comprehensive and efficient acquisition of Weibo data. (2) in the aspect of data mining, The user analysis module and Weibo analysis module of Weibo data mining are designed and implemented, which provides the basic functions of Weibo analysis, and focuses on the emotional analysis of Weibo text based on machine learning algorithm. The training experiment of classifiers is designed and implemented. In this paper, three feature extraction models, "word", "double word" and "word double word combination", are used to select the features by using chi-square statistical algorithm, and naive Bays is used. Six classification algorithms, such as logical regression and support vector machine, are compared. Through repeated experiments and comparison, the optimal classification model is obtained. The model has a good classification effect on Weibo text or shorter comment text. (3) in the aspect of data visualization, this paper adopts histogram, broken line chart, map, label cloud, cake chart, Dashboard and other visual charts show the results of data analysis, and presented through the browser. The system adopts B / S structure, the front end adopts browser to display the analysis results, and the background is composed of Weibo crawler, MySQL relational database and data mining module to realize the function of data collection, processing and analysis. Finally, the data mining and visual analysis of Weibo are realized. The main contributions and innovations of this paper are as follows: (1) A Weibo analysis system including Weibo data acquisition, data mining and data visualization is designed and implemented. The system realizes the functions of user analysis and Weibo analysis, and provides a basic platform for follow-up research. (2) the system realizes the emotional analysis function of Weibo text. In this paper, the machine learning algorithm is used to train the emotional analysis model. The correct rate of the model is 85%, and the AUC value is 0.94. The system can directly call the classifier to realize the emotional analysis of Weibo text.
【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092

【相似文献】