基于大数据的热点舆情发现与分析系统的设计与实现

发布时间：2018-06-03 11:01

本文选题：大数据 + Hadoop　；参考：《哈尔滨工业大学》2017年硕士论文

【摘要】：现实新闻内容生产环境中存在很多瓶颈,制约着新闻内容的生产。比如:短期热点无法捕捉、编辑人力有限、相关素材难以搜集、对已发表的报道缺乏合理的反馈机制。媒体需要一个能为他们及时发现热点,提供素材支持,追踪热点的工具。互联网已成为思想文化信息的集散地和社会舆论的放大器,这就使得舆情监控对企业、组织、机构而言非常重要。热点舆情发现与分析系统使用Hadoop计算平台分析大数据。Hadoop计算平台主要进行热点挖据和舆情分析。热点挖掘通过对一段时间内新闻数据进行挖掘,发现热点话题。舆情分析,对已挖掘的热点话题,将评论数据和社交数据与热点进行关联,通过情感分析、观点计算和用户画像进行舆情分析。所有数据使用Hadoop存储平台进行存储,对新闻数据建立索引,使用检索系统提供素材检索服务。最终,整个以网页形式程序呈现,为媒体写作提供热点发现与线索管理功能,为企业、组织、机构提供舆情分析和报警功能。系统通过下载平台从外网进行新闻与评论数据和新浪微博数据采集,通过内网内部推送流程进行社交和搜索数据采集。然后,系统对新闻数据和评论数据进行预处理,预处理主要包含地域分类、领域分类、低质量过滤、情感分析、站点识别和权威媒体认证。之后,一份数据将存储到Hadoop集群中被热点挖据流程使用,另一份将建立索引存储,索引数据可以被用作舆情分析也可以进行素材检索。之后,使用算法组件进行热点挖据和舆情分析,得到热点话题和舆情相关数据。算法组件主要包含热点挖据、热词发现、情感分析、观点计算和用户画像。前后台使用Hadoop文件和MySQL数据库,进行数据交互。最终,利用网页形式,根据不同的业务需求呈现数据。热点舆情发现与分析系统1.0版本已经完成人民日报的验收并获得肯定。当然,系统还有需要完善的地方。
[Abstract]:There are many bottlenecks in the production environment of news content, which restricts the production of news content. For example, short-term hot spots can not be captured, editors have limited manpower, relevant materials are difficult to collect, and there is no reasonable feedback mechanism for published reports. The media needs a tool to spot hot spots, provide material support, and track hot spots in time. The Internet has become the center of ideological and cultural information and the amplifier of public opinion, which makes monitoring of public opinion very important for enterprises, organizations and institutions. The hot spot public opinion discovery and analysis system uses the Hadoop computing platform to analyze the big data. Hadoop computing platform mainly carries on the hot spot digging and the public opinion analysis. Hot spot mining finds hot topics by mining news data for a period of time. Based on the analysis of public opinion, the comment data and social data are associated with the hot spots, and the public opinion is analyzed through emotional analysis, viewpoint calculation and user portrait. All the data are stored on the Hadoop storage platform, the news data is indexed, and the material retrieval service is provided by the retrieval system. Finally, the whole program is presented in the form of web pages, which provides hot spot discovery and clue management function for media writing, and provides public opinion analysis and alarm function for enterprises, organizations and institutions. The system collects news and comment data and Sina Weibo data from outside network through downloading platform, and social and search data collection through internal push flow of intranet. Then, the system preprocesses the news data and comment data. The preprocessing mainly includes regional classification, domain classification, low-quality filtering, emotional analysis, site identification and authoritative media authentication. After that, one piece of data will be stored in the Hadoop cluster and used by the hot spot collection process, and the other will be indexed. The index data can be used for public opinion analysis or for material retrieval. After that, the algorithm component is used to analyze hot spot and public opinion, and the data of hot topic and public opinion are obtained. The algorithm component mainly includes hot spot data, hot word discovery, emotion analysis, viewpoint calculation and user portrait. The front and back uses the Hadoop file and the MySQL database, carries on the data interaction. Finally, using the form of web pages, according to different business requirements to present the data. Hot public opinion discovery and analysis system version 1. 0 has completed the acceptance of People's Daily and has been confirmed. Of course, the system needs to be improved.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：G252.7;TP311.13

【参考文献】