WEB信息整合平台设计与实现

发布时间：2018-09-17 08:11

【摘要】：随着Internet技术的高速发展,网络信息资源的快速增长,网络已成为人们获取数据的重要来源。面对庞大的网络资源,搜索引擎为人们检索提供重要的技术手段。然而,传统的搜索引擎是基于单词的检索,存在一定的局限性,如搜索结果存在大量无关的网页、由于转载而造成信息内容雷同等。因此,极有必要对网络信息资源进行整合,以帮助人们从海量网络资源中,提炼出人们所关心的特定信息,并对数据重新整合与统一的展现。本文的主要研究工作就是整合WEB资源信息,使互联网用户能够快速准确地搜寻到自己需要的信息。首先,本文对WEB信息整合中的相关理论和技术研究,包括信息整合两种方法、三大组成模块以及四种关键技术等。并在设计过程中对各模块涉及知识做全面综述,包括本体概念、网络爬虫、信息抽取、资源描述框架等。其次,本文设计并实现了一种WEB信息整合平台原型系统,该系统以本体为指导。设计了系统总体结构框架模型,系统由4大模块组成：数据采集、信息抽取、存储模型、前台呈现。提出了基于本体和搜索引擎聚焦网络爬虫,基于本体的页面分析过滤算法,基于本体和DOM树路径的信息抽取规则,以及基于RDF的数据存储模型和基于B/S前台结果呈现等一系列设计方案。通过该信息整合平台,用户可以设置需要整合的领域信息,系统能够检索并整合出互联网中相关领域资源,并将结果以统一的、结构的、形象的展示给用户。该系统不需要对不同数据源分别建立包装器,而是作用域整个互联网之上,能够融合互联网中多种异构资源。最后,本文还对WEB信息整合平台做了综合测试,包括爬虫的效率与抓取量测试、数据抽取率测试等。测试证明系统能整合互联网中部分异质的数据源,但也存在一些不足。
[Abstract]:With the rapid development of Internet technology and the rapid growth of network information resources, the network has become an important source for people to obtain data. Facing the huge network resources, the search engine provides the important technical means for people's retrieval. However, the traditional search engine is based on word retrieval, there are some limitations, such as search results have a large number of unrelated web pages, because of reprinting and resulting in the same information content and so on. Therefore, it is very necessary to integrate the network information resources to help people extract the specific information that people care about from the massive network resources, and to reintegrate and unify the data. The main research work of this paper is to integrate WEB resource information so that Internet users can quickly and accurately search for the information they need. Firstly, this paper studies the theory and technology of WEB information integration, including two methods of information integration, three modules and four key technologies. In the process of design, the knowledge involved in each module is summarized, including ontology concept, web crawler, information extraction, resource description framework and so on. Secondly, this paper designs and implements a prototype system of WEB information integration platform, which is guided by ontology. The system is composed of four modules: data acquisition, information extraction, storage model and foreground presentation. This paper proposes a web crawler based on ontology and search engine, a page analysis filtering algorithm based on ontology, and information extraction rules based on ontology and DOM tree path. And a series of design schemes, such as data storage model based on RDF and foreground result presentation based on B / S, etc. Through the information integration platform, the user can set up the domain information that needs to be integrated. The system can retrieve and integrate the related domain resources in the Internet, and display the results to the user in a unified, structured and vivid way. The system does not need to set up wrappers for different data sources separately, but the scope of the entire Internet, and can integrate a variety of heterogeneous resources in the Internet. Finally, the paper also makes a comprehensive test on WEB information integration platform, including crawler efficiency and crawl test, data extraction rate test and so on. The test shows that the system can integrate some heterogeneous data sources in the Internet, but there are some shortcomings.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.1

【参考文献】