公众论坛信息实时检索的研究与实现

发布时间：2018-01-19 00:13

本文关键词： 公众论坛垂直搜索网络爬虫实时检索 Lucene　出处：《南京理工大学》2012年硕士论文　论文类型：学位论文

【摘要】：互联网作为当代社会迅猛发展的新生事物之一,已经开始扮演越来越重要的角色。公众论坛是互联网发展的产物之一,它是一个开放的平台,与普通网站的区别之处在于网民不仅可以通过它获取信息,同时也能发布信息,这为相互间的交流带来了极大的方便。但随着时间的发展,其产生的消极、危险的一面也渐渐显露出来——一些不法分子利用论坛的便利性散播各种非法信息。由于论坛中信息传播速度快,刷新频率高,非法信息很容易在短时间内就引起很严重的后果,因此需要及时的发现这些信息。本文设计了一个面向论坛领域的垂直搜索引擎,能够对指定论坛进行深度数据挖掘以及24小时监控新出现的信息。本文设计的垂直搜索引擎共分为三大模块：信息获取模块、信息分析模块、信息索引及检索模块。信息获取模块通过开发现有通用搜索引擎接口构建元搜索引擎以及编写网络爬虫实现；信息分析模块通过使用模板及网页信息去噪的方法实现了对HTML和Word、Excel、PDF等常见格式文件结构化文本信息的提取；信息索引和检索模块通过开源工具Lucene构建,为用户提供了便利高效的查询界面。用户使用反馈表明本文设计的垂直搜索引擎在深度数据挖掘以及实时监控方面都有着很好的性能。
[Abstract]:As one of the new things in the contemporary society, the Internet has begun to play an increasingly important role. Public forum is one of the products of the development of the Internet, it is an open platform. The difference with ordinary websites is that Internet users can not only get information through it, but also can publish information, which brings great convenience for mutual communication. But with the development of time, it produces negative. The dangerous side is also gradually revealed-some lawless elements use the convenience of the forum to spread all kinds of illegal information. Because of the rapid dissemination of information in the forum, refresh high frequency. Illegal information is easy to cause serious consequences in a short period of time, so we need to find these information in time. This paper designed a vertical search engine for forum domain. Ability to perform deep data mining and 24-hour monitoring of emerging information on specified forums. The vertical search engine designed in this paper is divided into three modules: information acquisition module, information analysis module. Information index and retrieval module. The information acquisition module constructs the meta search engine by developing the existing universal search engine interface and implements the web crawler. The information analysis module uses template and web page information denoising method to realize the extraction of structured text information of common format files such as HTML and WordWare Excel PDF. The information index and retrieval module is constructed by open source tool Lucene, which provides a convenient and efficient query interface for users. User feedback shows that the vertical search engine designed in this paper has good performance in depth data mining and real-time monitoring.
【学位授予单位】：南京理工大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.3

【参考文献】