基于Hadoop的网络舆情监控平台设计与实现

发布时间：2018-01-15 13:06

本文关键词：基于Hadoop的网络舆情监控平台设计与实现　出处：《计算机技术与发展》2016年02期 　论文类型：期刊论文

【摘要】：文中设计并实现了一种基于Hadoop的网络舆情监控系统。该系统以HDFS作为底层存储系统,在其上构建基于HBase的分布式数据库对舆情信息进行统一存储管理。首先利用基于MapReduce的分布式网络爬虫进行数据抓取,以解决单机爬虫效率低、可扩展性差等问题;其次采用Canopy结合K-means的二次聚类算法,克服单一K-means聚类算法的不足,以提高文本聚类的效率和准确度;最后实现基于查询的话题追踪策略,对热点话题进行有效跟踪分析。仿真实验表明:Canopy-Kmeans聚类方法比传统K-means方法漏报率、误报率分别降低1.24%、0.09%,最小标准代价降低1.681%。系统通过提供可视化舆情分析报告,为企业或单位及时掌握舆情热点、制定舆情策略提供科学、系统的技术支持。
[Abstract]:This paper designs and implements a network public opinion monitoring system based on Hadoop, which takes HDFS as the underlying storage system. The distributed database based on HBase is constructed to store and manage the public opinion information. Firstly, the distributed web crawler based on MapReduce is used for data capture. In order to solve the problems of low efficiency and poor expansibility of single crawler. Secondly, Canopy combined with K-means clustering algorithm is used to overcome the shortcomings of single K-means clustering algorithm, in order to improve the efficiency and accuracy of text clustering. Finally, the topic tracking strategy based on query is implemented, and the effective tracking analysis of hot topics is carried out. The simulation results show that the miss rate of the K-means clustering method is higher than that of the traditional K-means method. The false alarm rate is reduced by 1.24and 0.09, the minimum standard cost is reduced by 1.681. by providing visual public opinion analysis report, the system can grasp the hot spot of public opinion in time for enterprises or units. Formulate public opinion strategy to provide scientific and systematic technical support.
【作者单位】：山东省科学院情报研究所;
【基金】：山东省科学院青年基金项目(2013QN036) 山东省科技发展计划(2013GGX10127,2014GGX101013)
【分类号】：TP391.1
【正文快照】： 1概述控信息源,并结合传统的数据挖掘算法对获取的数据随着信息技术以及互联网的快速发展,其产生的进行文本统计与分析。基于流量镜像方法虽然可以比海量、异构、动态的新闻数据使得人们很难快速、高效较全面地收集各种网络信息,但也存在成本高、可扩展地找到用户感兴趣的新闻

【相似文献】