分布式元搜索数据采集系统的设计与实现

发布时间：2018-04-19 09:58

本文选题：信息爆炸 + 爬虫技术　；参考：《华中科技大学》2014年硕士论文

【摘要】：网络媒体的普及，提供了越来越多的信息发布平台。每天，各色各样的网站都在源源不断地制造新的内容，这些内容通过网络的形式在人群中四处传播。信息产生的速度已经远远超过人们消费它的速度。由此带来的结果是，信息无处不在，但是人们却缺乏一种行之有效的收集方式，大量数据并未被加以利用而浪费掉。这种信息四处泛滥而人们却无法有效获取到自己需要信息的矛盾将长期存在着。庆幸的是，爬虫技术的出现和搜索引擎的普及，可以极大地缓解这类矛盾。本文研究与实现的是一套分布式的元搜索数据采集系统。它的目标在于快速大量地获取到涉及产品质量安全的网络信息，从而为进行质量安全的预警与监控提供数据分析的基础。它基于Master/Slave的主从分布式架构，主控节点与采集节点有着不同的分工：主控节点主要负责抓取任务的定义与调度管理，采集节点负责网页数据的请求与获取。节点间通过网络通信进行数据的交换，主控节点将定义好的任务按照一定的调度策略分配给不同的采集节点，采集节点收到分配的任务之后，，启动爬虫程序下载网页，然后通过XQuery模板对获得的网页进行解析，再将结构化信息统一保存于HBase数据库中。同时，采集节点会对自己的运行状态进行统计，并提供外部接口给主控节点进行调用，从而实现对下载行为的控制。考虑到网页数据的分散性以及抓取成本的控制，本文提出采用元搜索技术手段，将现有的主流搜索引擎的数据作为本次数据抓取的入口，通过综合不同引擎的数据，为用户提供更全面更有针对性的数据。本文最后完成系统的搭建与部署，整个程序运行稳定，通过一些实验对系统的功能特性以及数据采集情况进行测试，并对结果进行了分析，从而验证了系统的可行性，同时对系统存在的不足提出改进建议。
[Abstract]:With the popularity of network media, more and more information publishing platforms are provided.Every day, a wide variety of websites create new content that spreads through the Internet.Information has been generated much faster than people consume it.As a result, information is everywhere, but people lack an effective way to collect it, and a lot of data is not used and wasted.This kind of information flooding and people can not effectively access to their own information will exist for a long time.Fortunately, the emergence of crawler technology and the popularity of search engines, can greatly alleviate such contradictions.This paper studies and implements a distributed meta-search data acquisition system.Its goal is to obtain network information about product quality and safety quickly and in large quantities, thus providing the basis of data analysis for early warning and monitoring of quality and safety.It is based on the master-slave distributed architecture of Master/Slave. The master node and the acquisition node have different division of labor: the master node is mainly responsible for the definition and scheduling management of the grab task, and the acquisition node is responsible for the request and acquisition of the web page data.The main control node distributes the defined tasks to different collection nodes according to a certain scheduling strategy. After receiving the assigned task, the acquisition node starts the crawler program to download the web page.Then the web pages are parsed by XQuery template, and the structured information is stored in the HBase database.At the same time, the collection node will count its own running state, and provide external interface to the main control node to call, so as to control the download behavior.Considering the dispersion of web page data and the cost control of crawling, this paper proposes to use meta-search technology to use the existing mainstream search engine data as the entry point of this data capture, by synthesizing the data of different engines.Provide users with more comprehensive and targeted data.Finally, the system is constructed and deployed, and the whole program runs stably. The functional characteristics and data acquisition of the system are tested through some experiments, and the results are analyzed to verify the feasibility of the system.At the same time, some suggestions to improve the system are put forward.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP274.2;TP391.3

【相似文献】