针对动态网络数据的分布式增量获取方法

发布时间：2018-10-09 11:14

【摘要】：伴随着互联网的爆炸性发展,Web已经发展成为站点遍布全球的巨大信息服务网络,越来越多的人也开始把自己的精力和时间都投向了互联网。电商、视频、论坛、微博等都是获取重要数据的来源。无论是自己想要能够及时获取最新的数据来进行研究亦或者是需要批量获取大量的数据来进行相关分析,都需要一个适当高效的方法来支持这些数据的提取。爬虫技术作为大规模提取数据的技术之一,又重新了映入人们的眼帘。人们在使用爬虫技术从互联网上获取大量的数据的同时,也在不断对其做出改进和完善。在经过对爬虫技术、分布式技术、Docker技术、以及Linux相关技术的学习之后,本论文中总计完成了以下工作:首先提出了实用爬虫的设计思路,并重点详细阐述了信息提取模块和数据存储模块的设计与实现方法以及如何解决网站反爬问题。随后搭建了基于分布式爬虫框架PySpider的分布式爬虫集群来代替人工手动分布式爬虫,并解决了搭建集群时遇到的一些Linux服务器相关问题。接着为了改进分布式爬虫框架PySpider的运行效率,设计并实现了一种使用Docker来构建分布式爬虫集群的方法。对改进前后的爬取效率进行了测试,并对测试结果做出了相应的分析。最后为了结合项目中的实际工作需要,将增量爬虫和分布式爬虫框架PySpider进行了结合,最终实现了基于动态网络数据的分布式增量获取方法。在本文的最后,对目前的工作做出了总结,并对未来的工作进行了相应的展望和部署。
[Abstract]:With the explosive development of the Internet, Web has developed into a huge information service network with sites all over the world, and more people have begun to invest their energy and time on the Internet. E-commerce, video, forums, Weibo and so on are important sources of data. Whether they want to be able to obtain the latest data in time for research or need to obtain a large number of data for correlation analysis, they need a suitable and efficient method to support the extraction of these data. Reptile technology, as one of the large-scale data extraction techniques, has reappeared in people's eyes. While using crawler technology to obtain a large amount of data from the Internet, people are constantly improving and perfecting it. After the study of crawler technology, distributed technology and Linux related technology, the following works have been accomplished in this paper: firstly, the design idea of practical reptile is put forward. The design and implementation of the information extraction module and the data storage module as well as how to solve the backcrawling problem of the website are described in detail. Then, a distributed crawler cluster based on distributed crawler framework (PySpider) is built to replace manual distributed crawler, and some problems related to Linux server are solved. Then, in order to improve the efficiency of distributed crawler framework (PySpider), a method of constructing distributed crawler cluster using Docker is designed and implemented. The crawling efficiency before and after improvement was tested and the test results were analyzed. Finally, in order to meet the practical needs of the project, the incremental crawler and the distributed crawler framework (PySpider) are combined. Finally, the distributed incremental acquisition method based on dynamic network data is realized. At the end of this paper, the current work is summarized, and the future work is prospected and deployed.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092

【相似文献】