网络爬虫性能提升与功能拓展的研究与实现

发布时间：2018-07-22 12:20

【摘要】：随着网络的迅速发展,万维网成为大量信息的载体,如何有效地提取并利用这些信息成为一个巨大的挑战。为满足这种需求,网络爬虫应运而生。它是一种按照一定的规则,自动的抓取万维网信息的程序或者脚本。首先,本文介绍了网络爬虫的发展历史及其应用领域,通过分析主流的网络爬虫,发现当今的网络爬虫主要服务于搜索引擎,为面向主题的用户查询准备数据资源。从网络爬虫的拓展性极强的爬行架构出发,传统爬虫对于搜索引擎的重要性逐渐淡化了它在灵活性和功能性上的特点。然后,本文探讨了评价网路爬虫性能的若干指标,并以此出发,从性能提升和功能拓展两个方面详细介绍了中小型网络爬虫的优化策略。在性能提升方面,本文根据不同的功能模块分别介绍了若干优化方案。第一,选择Gzip/deflate压缩编码传输,通过降低传输量来降低网络传输时间；第二,异步请求下载,提高带宽占用率和CPU利用率；第三,采用广度优先爬行,使用布隆过滤器实现大规模的URL去重检测；第四,采用设计精细的正则表达式提取页面链接；第五,对爬取的URL进行严格的正规化处理,降低URL的出错对爬虫的误导性；第六,委托优化后的线程池高效率地管理多线程。在功能拓展方面,本文主要从以下三方面进行区别于传统爬虫的新尝试。第一,静态页面性能分析,给网站提供性能改进意见；第二,充当自动化测试工具,用于在指定页面执行测试用例；第三,可定制的聚焦数据提取,根据用户的需求进行指定格式的数据抓取。基于验证上述优化策略的目的,.NET平台特别适合架构轻量级的爬虫。该爬虫采用.NET平台下的C#语言,在Visual Studio2008环境下进行开发。程序在命令行模式下运行,具有基于文件的高可配置性。
[Abstract]:With the rapid development of the network, the World wide Web has become the carrier of a lot of information. How to extract and utilize the information effectively becomes a huge challenge. In order to meet this demand, the network crawler came into being. It is a program or script that automatically grabs World wide Web information according to certain rules. Firstly, this paper introduces the history of web crawler and its application field. By analyzing the mainstream web crawler, it is found that today's web crawler mainly serves search engine and prepares data resources for subject oriented user query. Based on the highly extensible crawling architecture of web crawlers, the importance of traditional crawlers to search engines has gradually weakened its flexibility and functional characteristics. Then, this paper discusses some indexes to evaluate the performance of web crawlers, and then introduces the optimization strategies of small and medium-sized web crawlers from two aspects of performance improvement and function expansion. In terms of performance improvement, this paper introduces several optimization schemes according to different function modules. First, choose Gzip-deflate compression code transmission to reduce the network transmission time by reducing the amount of transmission; second, asynchronous request download, increase bandwidth utilization and CPU utilization; third, use breadth first crawling, Using Bloom filter to achieve large-scale URL re-detection; fourth, using well-designed regular expressions to extract page links; fifthly, strictly regularizing the URL crawled to reduce the error of URL to the reptile misleading; sixth, The optimized thread pool efficiently manages multithreading. In the aspect of function expansion, this paper mainly tries to distinguish the traditional reptile from the following three aspects. First, static page performance analysis provides performance improvement advice to the website; second, it acts as an automated test tool for performing test cases on a specified page; third, customizable focused data extraction, According to the needs of the user for the specified format of data capture. Based on the verification of the above optimization strategy, the .NET platform is particularly suitable for lightweight crawlers. The crawler is developed in Visual Studio 2008 with C # language based on. Net platform. The program runs in command-line mode and is highly configurable based on files.
【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.3

【参考文献】