垂直搜索引擎中智能爬虫系统的研究与实现

发布时间：2018-12-22 08:42

【摘要】：随着信息技术的发展,搜索引擎已经成为了互联网的入口。垂直搜索引擎在垂直领域深度采集数据并加工处理,提供精准和专业搜索服务,具有广泛的关注度和市场需求。智能爬虫是垂直搜索引擎的重要组成部分,完成数据采集的工作。由于其专业背景强,覆盖范围窄,不同的智能爬虫在架构和策略上差异性很大,同时也面临主题相关性的精确度、小范围的大量采集困难等问题。本文针对垂直搜索引擎中的智能爬虫的技术难点,研究相关技术,提出了解决方案,在插件化和分布式的设计原则下,设计和实现了一套完整的智能爬虫系统,同时对系统的功能和性能进行了测试。本文的主要工作包括:(1)设计了一个基于LDA的文本特征提取方法,用机器学习算法决策主题相关性,并建立了以锚文字特征和页面主题相关度为基础的链接预测模型。(2)设计了多策略的应对反爬虫方案和代理服务器筛选流程。(3)提出了基于Bloomfilter的两层三实例的高可用,持久化URL去重方案,解决了海量URL的快速精准去重。(4)完成智能爬虫其它功能的设计,编码实现了完整的系统。通过搭建实验拓扑环境并部署爬虫系统,验证和测试了智能爬虫系统的功能和性能。试验结果表明,本文的设计和实现达到预期目标,提升了爬虫的智能性和效率。
[Abstract]:With the development of information technology, search engine has become the entrance of the Internet. Vertical search engine collects data deeply in vertical field and processes it to provide accurate and professional search service with wide attention and market demand. Intelligent crawler is an important part of vertical search engine. Because of its strong professional background, narrow coverage, different intelligent reptiles have great differences in architecture and strategy, but also face the accuracy of thematic correlation, a large number of collection difficulties in small areas and so on. Aiming at the technical difficulties of intelligent crawler in vertical search engine, this paper studies the related technologies and puts forward a solution. Under the principle of plug-in and distributed design, a complete intelligent crawler system is designed and implemented. At the same time, the function and performance of the system are tested. The main work of this paper is as follows: (1) A text feature extraction method based on LDA is designed. A link prediction model based on anchor text features and the relevance of page themes is established. (2) A multi-strategy anti-reptilian scheme and proxy server filtering process are designed. (3) the high availability of two layers and three instances based on Bloomfilter is proposed. Persistent URL reshuffling scheme solves the fast and accurate weight removal of mass URL. (4) the design of other functions of intelligent reptiles is completed and the complete system is coded. The function and performance of the intelligent crawler system are verified and tested by setting up the experimental topology environment and deploying the crawler system. The experimental results show that the design and implementation of this paper reach the expected goal and improve the intelligence and efficiency of reptiles.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【相似文献】