基于Solr的企业级检索系统的设计与实现

发布时间：2018-02-04 01:53

本文关键词： 企业级搜索引擎分布式一致性哈希 SOLR　出处：《华南理工大学》2013年硕士论文　论文类型：学位论文

【摘要】：搜索引擎是一项伟大的技术，它使人们从浩如烟海的网页中解放出来。企业级搜索引擎是面向企业应用的中小型搜索引擎，可帮助企业处理内部信息，并将各种企业相关的网络信息联系起来，实现资源的共享及整合。木棉检索是面向校园网应用的企业级搜索引擎，也是下一代互联网分布式搜索平台SE6的主要节点搜索引擎。本文在其原有架构的基础上，对一些核心模块及流程做了重新设计，并加入一些新的模块，使系统在性能、扩展性、容错性等能力上有更大的提高。本文针对查询性能的优化，重新设计了查询模块，搜索节点引入了开源企业级搜索引擎——Solr，并设计了分布式网页存储，以一致性哈希为划分策略。在保持原有系统并行查询的特点外，加入了对索引的维护功能，包括增、删、改索引；节点通信方式也由RPC改成了更加开放、标准的HTTP方式，，接口更加规范。重新设计后，系统的查询效率得到了提高，开放、扩展性也得到了提升。针对正文管理不规范、生成摘要速度慢、索引冗余等问题，本文设计了网页元数据管理系统。与原有的正文管理方式相比，网页元数据管理更加系统、规范、高效，满足了网页规模不断增长的需求，在存储节点增、删时，能快速重新划分及完成数据迁移的工作。为了提高系统的容错性、扩展性和错误恢复能力，本文设计了动态发现机制。动态发现机制抛弃了原有的节点管理方式，分布式系统里的节点分布等信息统一由动态发现机制维护。通过动态发现机制，在节点新增、宕机、退出、网络异常等情况下，系统依然能保持正常的服务状态，容错能力大大提高。本文最后对整个系统进行了性能评测。评测主要通过索引的建立速度、网页在节点间的分布是否均匀、查询响应速度几个方面进行，并通过与原有系统的对比来评测最终效果。测试的数据来自实验室SE6分布式搜索引擎平台中的校园网在线数据。
[Abstract]:Search engine is a great technology, it liberates people from the vast web pages. Enterprise search engine is a small and medium-sized search engine for enterprise applications, which can help enterprises handle internal information. And related to various enterprises related to network information to achieve the sharing and integration of resources. Kapok Retrieval is an enterprise-level search engine for campus network applications. It is also the main node search engine of the next generation Internet distributed search platform SE6. Based on its original architecture, this paper redesigns some core modules and processes, and adds some new modules. Make the system in the performance, expansibility, fault-tolerant and other capabilities have a greater improvement. Aiming at the optimization of query performance, this paper redesigns the query module, introduces open source enterprise-class search engine Solr, and designs distributed web storage. In addition to maintaining the characteristics of parallel query in the original system, the maintenance function of the index is added, including adding, deleting and changing the index. The node communication mode is changed from RPC to more open standard HTTP mode and the interface is more standardized. After redesigning the query efficiency of the system has been improved open and expansibility has also been improved. Aiming at the problems of non-standard text management, slow summary generation and index redundancy, this paper designs a web page metadata management system. Compared with the original text management mode, web metadata management is more systematic and standardized. It can quickly repartition and complete the work of data migration when the storage nodes increase and delete. In order to improve the fault tolerance scalability and error recovery ability of the system. This paper designs the dynamic discovery mechanism. The dynamic discovery mechanism abandons the original node management mode, and the information of node distribution in the distributed system is maintained by the dynamic discovery mechanism. In the case of new nodes, outages, exits, network anomalies, the system can still maintain a normal service state, and the fault-tolerant ability is greatly improved. At the end of this paper, the performance of the whole system is evaluated. The evaluation is mainly carried out through the speed of building index, the distribution of web pages among nodes, and the response speed of query. Finally, the final result is evaluated by comparing with the original system. The test data come from the campus network online data in the laboratory SE6 distributed search engine platform.
【学位授予单位】：华南理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【参考文献】