分布式信息检索系统的优化设计和实现

发布时间：2019-03-28 11:46

【摘要】：传统搜索引擎采用集中式的信息爬取和索引方式,对网络上深层内容、动态内容和私有内容的处理具有一定局限性。分布式信息检索则能够更好的适应对各种异构资源的检索。它可以对多种来源的信息进行有效的整合和处理,提供更为多样化的检索服务。其检索流程可分为资源描述、资源选择、查询分发和结果融合四个阶段。其中查询分发阶段会伴随着大量的网络通信,使用一般的IO模型和通信方式会造成较大的开销,因此需要进行特殊的设计。此外,检索系统还需要具备良好的可扩展性,以便处理异构的资源和多样化的查询需求。服务注册、服务管理、服务发现和服务监控等基础设施对于分布式系统的稳定运行也起着关键的作用。本文的工作重点即是设计和实现一个高效、稳定、可扩展的分布式信息检索系统。主要包括以下几个部分:(1)分布式信息检索系统的整体架构设计,根据分布式信息检索的特点,划分系统的功能模块,分析和选择系统所使用的IO模型和通信方式。(2)设计和实现系统的服务注册、服务管理、服务发现和服务监控等基础组件,保证系统各节点服务间的访问和通信能够稳定、可靠的进行。(3)在系统的核心检索模块中,定义资源选择、查询分发和结果融合的接口,并实现相应的算法。设计和实现插件机制,以支持对核心检索模块算法和功能的灵活扩展。实现资源选择和查询分发的缓存功能,以提升系统的吞吐量,减少查询的响应时间,节约带宽。(4)建立中央抽样库,保存各个资源库的抽样文档,以支持资源选择过程。实现查询抽样工具,基于资源库的检索接口,对资源库进行查询抽样,并将结果导入中央抽样库中。(5)对系统的功能和性能进行测试,比较和分析系统在不同查询参数、资源库响应时间、资源库数量以及不同并发数下的性能变化情况。
[Abstract]:The traditional search engine adopts a centralized information crawling and indexing method, and has certain limitation on the deep content, dynamic content and the processing of the private content on the network. The distributed information retrieval can be better adapted to the retrieval of various heterogeneous resources. It can effectively integrate and process the information of a variety of sources, and provide more diversified interactive services. The retrieval process can be divided into four stages: resource description, resource selection, query distribution and result fusion. In which the query distribution stage is accompanied by a large number of network communication, and a general IO model and a communication mode can be used to cause a large overhead, so a special design is required. In addition, the retrieval system needs to have good scalability to handle heterogeneous resources and diverse query requirements. Infrastructure for service registration, service management, service discovery and service monitoring also plays a key role in the stable operation of distributed systems. The focus of this paper is to design and implement a highly efficient, stable and scalable distributed information retrieval system. The paper mainly includes the following parts: (1) The overall architecture design of the distributed information retrieval system, according to the characteristics of the distributed information retrieval, the functional module of the division system, the IO model and the communication mode used by the analysis and selection system. (2) The basic components such as service registration, service management, service discovery and service monitoring of the system shall be designed and implemented so as to ensure the access and communication among all the service nodes of the system to be stable and reliable. And (3) in the core search module of the system, a resource selection, a query distribution and a result fusion interface are defined, and a corresponding algorithm is realized. The plug-in mechanism is designed and implemented to support a flexible extension of the core search module algorithm and functionality. And the cache function of resource selection and query distribution is realized, so that the throughput of the system is improved, the response time of the query is reduced, and the bandwidth is saved. (4) establishing a central sampling bank, and storing the sampling documents of each resource library so as to support the resource selection process. The query sampling tool is implemented, the resource pool is queried and sampled based on the retrieval interface of the resource pool, and the result is imported into the central sampling library. (5) The function and performance of the system are tested, and the performance changes of the system under different query parameters, resource pool response time, resource pool quantity and different concurrent numbers are compared and analyzed.
【学位授予单位】：华南理工大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.3

【相似文献】