基于J2EE的网络舆情分析系统的设计与实现

发布时间：2018-09-19 18:42

【摘要】：随着网络信息的极速膨胀,各类新闻网站,论坛上的信息越来越庞大。在如此庞大的数据中,快速且准确的找到和用户关心的主题有关的新闻条目或是论坛主题变得越来越困难。目前Google、百度这样的搜索引擎提供的爬取结果的主题过于广泛,时效性也难以保证,加上一些和关键词相关的结果位置被购买,往往导致检索的时效性,相关性,人工检索的效率皆不如人意,满足不了特定领域用户的精准搜索的需要。为此本文的目的就是实现一个精准的对页面时效性,网络内容相关性的和搜索关键词高度匹配的网络爬虫系统,加之准确的分类分析,以达到网络舆情监控的目的。随着互联网治安越来越成为构建和谐精神文明建设的一个重要组成部分,对网络水军与网络敌对势力的防范,对来自人民的声音的听取变得越来越重要。而这也导致了国内目前做舆论监控为目的的网络爬虫系统的公司层出不穷,本人实习所在单位就是这样一家专注于特定用户群的互联网公司。本文所描述的网络舆情分析系统的数据来源为目前国内主要微博、论坛(如西祠,天涯,新浪,贴吧)和新闻网站(新浪等门户),以及百度,奇虎新闻等专业搜索引擎的结果页面。网页递归地抓取完成后使用HtmlParser这样一个开源的网页解析工具解析出主要结构,进行基于时间,主题相关性,内容相关度的分类过滤,最后更新到数据库中供前台Jsp页面访问。爬虫支持定时任务与ad-hoc即时触发。逻辑层面使用了一些如Spring、Hibernate、Struts这样的开源技术来构建MVC-based的业务处理系统,后台的采集、解析、过滤等模块还使用了诸如Berkeley DB、Apache Lucene、HtmlParser、MMAnalyzer等开源技术来完善系统。在这个系统中本人设计并实现了采集中心子模块、解析模块以及分类分析模块。论文主要描述了这几大模块的相关工作,包括需求分析、具体设计(流程图、类图)以及关键代码的实现。最后描述了系统的缺点与待改进之处。
[Abstract]:With the rapid expansion of network information, all kinds of news websites and forums are getting more and more information. In such huge data, it is becoming more and more difficult to find news items or forum topics related to topics of concern to users quickly and accurately. At present, search engines such as Google, Baidu provide a wide range of topics for crawling results, and the timeliness is difficult to guarantee. In addition, some key words related to the results are purchased, which often leads to the timeliness and relevance of retrieval. The efficiency of manual retrieval is not satisfactory, which can not meet the needs of users in a specific field of accurate search. Therefore, the purpose of this paper is to achieve a precise web crawler system with the accuracy of page timeliness, network content correlation and high matching of search keywords, together with accurate classification and analysis, in order to achieve the purpose of monitoring network public opinion. As Internet public order becomes an important part of constructing harmonious spiritual civilization, it is more and more important to guard against the network naval forces and network hostile forces, and to listen to the voices of the people. This has also led to the emergence of domestic public opinion monitoring for the purpose of the network crawler system companies, my internship in the unit is such a focus on specific user groups of Internet companies. The data sources of the network public opinion analysis system described in this paper are currently the main domestic Weibo, forums (such as Xici, Tianya, Sina, Tiefa) and news websites (Sina and other portals), as well as Baidu, Qihoo News and other professional search engine results page. After the web page is fetched recursively, we use HtmlParser, an open source web page parsing tool, to analyze the main structure and filter it based on time, topic correlation and content correlation. Finally, it is updated to the database for the foreground Jsp page to visit. Reptiles support timing tasks and ad-hoc immediate triggers. At the logic level, some open source technologies such as Spring,Hibernate,Struts are used to build MVC-based business processing system, and the modules of background acquisition, parsing, filtering and so on are used to perfect the system by using open source technology such as Berkeley DB,Apache Lucene,HtmlParser,MMAnalyzer. In this system, I designed and implemented the collection center sub-module, analysis module and classification analysis module. This paper mainly describes the related work of these modules, including requirements analysis, specific design (flow chart, class diagram) and the implementation of key codes. In the end, the shortcomings of the system and the points for improvement are described.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.3

【相似文献】