基于Actor模型的广域分布式爬虫的设计与实现

发布时间：2018-05-10 23:15

本文选题：分布式 + 爬虫　；参考：《北京邮电大学》2016年硕士论文

【摘要】：计算机技术与网络技术的迅猛发展将我们带入了一个全新的互联网时代。网民们每天在互联网上可以产生大量的行为数据。本课题实现的基于Actor模型的广域分布式网络爬虫系统就是用于获取互联网上大量公开数据的一个基础框架系统。该系统通过搭建一套完整的分布式爬虫框架,为爬虫的开发提供了一体化的服务。得益于开源社区的活跃,目前已经有了一些比较成熟的开源爬虫系统,其中以Java版的Heritrix/Nutch和Python版的Scrapy较为被大家熟知。这些框架大都功能齐全,文档丰富,但是也存在一定的问题,在特定场景下,可能并不是最佳的选择。且由于长期的发展,其代码规模也十分庞大,在出现问题时较难诊断。另外,它们对于分布式的支持要么比较薄弱,要么依赖于其它分布式框架,在现今机器资源充足的条件下,显得比较单薄。在此背景下,本课题实现的基于Actor模型的广域分布式爬虫框架,是一个供爬虫开发者使用的系统。在该框架之上,开发者可以快速有效的完成一个分布式爬虫任务。本课题从实际的爬虫任务的功能和性能需求出发,制定了总体的设计方案,并以此为基础设计了一套完成的爬虫框架。从总体上看,本课题实现的系统分为五个模块,包括Master模块,Slave模块,Client模块,Worker模块和存储模块。其中,Master模块负责框架整体的运行;Slave模块负责在Slave节点建立运行进程;Worker模块负责具体页面的爬取解析和存储;Client模块负责作业的提交和运行时的操作管理;后端存储模块提供了数据的存储方式。各个模块在功能和运行态上相互独立,模块间的通信主要通过HTTP方式进行。在各个模块内部,又包含多个子模块。模块内各个子模块之间主要通过Actor模型进行消息通信。本文从详细设计与实现角度对系统进行了全面的剖析,并对系统进行了功能和性能测试。最后,本文对本课题完成的系统做了总结和展望,提出一些可行的改进方案。
[Abstract]:The rapid development of computer technology and network technology has brought us into a new Internet era. Internet users can produce a lot of behavior data on the Internet every day. The Wide-area distributed Web crawler system based on Actor model is a basic framework system for obtaining a large number of open data on the Internet. The system provides an integrated service for crawler development by building a complete distributed crawler framework. Thanks to the active open source community, there are some mature open source crawler systems, among which the Java version of Heritrix/Nutch and the Python version of Scrapy are well known. Most of these frameworks are fully functional and well documented, but there are some problems, which may not be the best choice in a particular scenario. And because of the long-term development, its code scale is very large, when the problem is difficult to diagnose. In addition, their support for distributed systems is either weak or dependent on other distributed frameworks. In this context, the Wide-area distributed crawler framework based on Actor model is a system for crawler developers. On top of this framework, developers can quickly and effectively complete a distributed crawler task. Based on the functional and performance requirements of the actual crawler task, the overall design scheme is formulated and a set of completed crawler framework is designed. On the whole, the system is divided into five modules, including Master module, client module and storage module. The master module is responsible for the running process of the Slave node and the storage client module is responsible for the submission of the job and the operation management of the run time. The module is responsible for establishing the running process in the Slave node and taking charge of the crawling and parsing of the page and the storage client module. The back-end storage module provides a way to store data. Each module is independent of each other in function and running state. The communication between modules is mainly carried out by HTTP. Within each module, there are several sub-modules. The message communication between each sub-module of the module is mainly carried out by Actor model. In this paper, the detailed design and implementation of the system from the point of view of a comprehensive analysis, and the system function and performance test. Finally, this paper summarizes and prospects the system completed in this paper, and puts forward some feasible improvement schemes.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.52

【相似文献】