面向特定领域的主题搜索系统的研究与实现

发布时间：2018-03-28 07:49

本文选题：主题搜索　切入点：网页过滤　出处：《北京邮电大学》2012年硕士论文

【摘要】：随着互联网信息量的爆炸式增长,网络中出现了大量的垃圾信息以及重复信息。通用的搜索引擎已经难以满足用户专业化、个性化的搜索需求,主题搜索应运而生。基于主题搜索的研究现状,本文对面向特定领域的主题搜索系统进行了深入的研究。目前互联网中有海量的政府企业的招标／中标信息,对于企业来说,及时地掌握国内招标信息意义重大,故本文将特定领域设为“招标”领域,研究并实现面向招标领域的主题搜索系统。论文主要研究工作和成果包括：第一,提出了基于双重特征选择的网页过滤方法。对CHI特征选择算法进行改进,提出了双重特征选择算法,并结合改进的TF-IDF公式,使用SVM分类器进行二元分类。实验表明,该方法具有更好的网页过滤效果。第二,提出了适用于招标领域站点的增量搜集模型。基于招标领域站点的七个特点,从增量搜集的对象、增量搜集的方法以及增量搜集的时间三个方面对模型进行阐述,并通过实验证明了该模型的有效性。第三,设计并实现了一个面向招标领域的主题搜索系统。本文具体对该系统的主题爬虫模块、网页过滤模块、网页解析模块、文本分类模块以及增量搜集模块进行了详细的设计与实现,并达到了较好的运行效果。本文对面向招标领域的主题搜索系统的研究,可以满足政府企业对最新招标／中标信息的需求,具有重大的现实意义。
[Abstract]:With the explosive growth of Internet information, there is a lot of spam and repeated information in the network. The general search engine has been difficult to meet the specialized and personalized search needs of users. Topic search emerges as the times require. Based on the current research situation of topic search, this paper makes a deep research on the topic search system oriented to specific fields. At present, there is a huge amount of bidding / winning information of government enterprises on the Internet, which is of great significance for enterprises to grasp domestic bidding information in a timely manner. Therefore, this paper designates a specific field as a "bidding" field. Research and implementation of the subject search system in the field of bidding. The main research work and achievements include:. Firstly, a method of web page filtering based on double feature selection is proposed. The CHI feature selection algorithm is improved, and the dual feature selection algorithm is proposed. Combined with the improved TF-IDF formula, SVM classifier is used for binary classification. This method has better web filtering effect. Secondly, an incremental collection model suitable for bidding site is proposed. Based on the seven characteristics of bidding site, the model is described from three aspects: the object of incremental collection, the method of incremental collection and the time of incremental collection. The validity of the model is proved by experiments. Thirdly, we design and implement a subject search system oriented to the bidding field. In this paper, the theme crawler module, the web filtering module, the web page analysis module are introduced. The text classification module and the incremental collection module are designed and implemented in detail, and good results are achieved. In this paper, the research on the subject search system in the field of bidding can meet the needs of the government enterprises for the latest bidding / winning information, which is of great practical significance.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.3

【参考文献】