基于主题的互联网信息抓取研究

发布时间：2018-03-17 01:21

本文选题：互联网信息抓取　切入点：主题爬虫　出处：《浙江大学》2014年博士论文　论文类型：学位论文

【摘要】：互联网信息抓取是利用计算机自动从互联网中获取信息的方法,在搜索引擎,情报收集系统中得到广泛应用。传统抓取技术主要研究对全网数据抓取策略的调优,以保证信息的新鲜度。但是全网抓取会导致信息处理深度不够,专业性不强,无法满足企业的信息需求；而且大多数中小型企业并没有足够的计算资源支持全网信息抓取,所以基于主题的抓取系统应运而生。随着聚焦爬虫概念的提出,基于特定主题的抓取系统研究开始受到研究者的关注。如何使用最少的计算资源,抓取到尽可能多的主题相关页面是主题抓取系统面临的主要挑战。其中涉及到三个关键的问题：1)如何定义用户主题：2)如何进行网页主题判断；3)如何预测待抓取URL的主题相关性,并制定相应的抓取策略。本文深入研究了基于主题的互联网信息抓取技术,分析了其中的关键问题,并提出了相应的解决方案,主要贡献如下：1)针对主题抓取系统的三个特点：主题需求的开放性,主题聚焦的层次性和主题信息的局部性,提出了基于主题知识库的互联网主题信息抓取框架。围绕主题知识库,提供了综合主题需求表达方式,知识学习流程和网页主题判断方法。通过主题富饶域挖掘模块进一步优化抓取效率。2)针对主题表达的开放性和动态性问题,提出了基于稳定词集的主题需求封闭流程,并在此基础上进一步提出了基于迭代式扩展-过滤框架的稳定词集构造方法。通过频繁项挖掘和LDA分析两种方法对核心主题词进行扩展,并使用知识库对扩展词集进行过滤。实验表明此方法获得的稳定词集具有较强的主题代表性。3)针对主题聚焦的层次性问题,提出了基于本体的网页主题判断算法,并应用于主题爬虫。利用本体中的概念以及概念间的位置关系,对网页信息进行主题降维,并通过本体综合加权提高主题判断的准确性,从而提高主题抓取的收获率。通过实验与其他的降维方法进行比较,证明了利用本体进行主题降维的有效性。4)针对互联网信息的主题局部性特点,提出了基于主题富饶域的抓取策略。主题富饶域优先(TRDF)算法根据主题浓度将主题域分为三个不同集合,针对不同集合采取差序化抓取策略。实验结果表明TRDF策略在准确率和召回率上均优于现有算法。
[Abstract]:Internet information capture is a method of automatically obtaining information from the Internet by using the computer, in the search engine, is widely used in intelligence collection system. The main research of traditional grasping technology tuning of the whole network data capture strategy, to ensure the freshness of information. But the whole network grasping leads to information processing deep enough, professional is not strong that can not meet the enterprise information needs; and the majority of small and medium enterprises and not enough computing resources to support the whole network information capture, so as grasping system based on topic. With the development of the concept of focused crawler, crawling system specific topics began to attention of researchers. Based on how to use the least amount of computational resources, to grasp the theme the relevant page as much as possible is the main challenge topic crawling system which involves three key questions: 1) how to define Family theme: 2) how to determine the theme "; 3) to predict the relevance to grab URL, and to develop the corresponding capture strategy. This paper studies the Internet information crawl technology based on topic, analyzes the key problems, and put forward the corresponding solutions. The main contributions are as follows: 1) three according to the characteristics of subject crawling system: opening theme demand, local focus levels and thematic information, put forward the theme of Internet information extraction framework based on subject knowledge. Around the theme of knowledge base, provides comprehensive requirements of expression, knowledge learning process and web page topic judgment method. Through the theme of rich domain mining module to further optimize the crawl efficiency.2) for theme expression of the open and dynamic problems, put forward the theme of stable word set demand closed process based on this base Based on the proposed iterative extended stable word filtering framework set construction method based on frequent item mining and LDA. Through the analysis of two methods to expand the core keywords, and use the knowledge base of extended word set filter. The experimental results show that the stability of this method to obtain a set of words with a strong theme of representative.3 aiming at the problem level) focus, put forward the algorithm to determine "theme based on ontology, and applied to the topic crawler. Using the position relationship between concepts in ontology and concept, subject to reduce the dimensionality of the web information, and through the body to improve the accuracy of judging the comprehensive theme, so as to improve the harvest rate. Grasping the theme by comparing with other experimental method of dimensionality reduction, proved the validity of.4 subject dimensionality reduction using ontology) theme local characteristics for the Internet information, is proposed based on the theme The Rao domain crawl strategy. The theme enriched area priority (TRDF) algorithm divides the topic domain into three different collections according to the topic concentration, and adopts the differential sequence crawling strategy for different sets. The experimental results show that the TRDF strategy is superior to the existing algorithm in accuracy and recall rate.

【学位授予单位】：浙江大学
【学位级别】：博士
【学位授予年份】：2014
【分类号】：TP391.1

【相似文献】