基于Hadoop的分布式搜索引擎的研究及实现

发布时间：2018-06-24 08:00

本文选题：分布式搜索引擎 + HITS　；参考：《电子科技大学》2017年硕士论文

【摘要】：今天互联网已经发展到了大众普遍参与的时代,人们使用网络的方式愈发多样,发布信息也愈发方便,这使得网络中充斥着海量的数据。怎样对这些海量数据进行索引和检索是目前搜索引擎研究的重点。显然,集中式索引已经无法满足当前大数据环境下的要求。为此,人们将分布式技术与索引技术进行结合,实现了分布式索引和检索。在分布式索引中采用怎样的索引策略对索引和检索的效率起到了关键性的影响,因此索引策略具有重要的研究价值。搜索引擎会获取到各种各样的数据,当用户搜索时,会检索出大量的结果页面。如何将搜索结果中的网页按照其重要程度进行排序,使得相对于用户查询最重要的网页排到输出结果的最前面,节省用户查找信息的时间是一项很有意义的研究工作。因此从上述角度考虑,本文对分布式索引策略和链接排序算法进行了研究。主要做了如下工作:1.对分布式搜索引擎相关理论进行了学习研究,重点对分布式的索引策略进行了研究,包括局部索引策略和全局索引策略,提出了基于Map Reduce和DHT的混合索引策略,并对Map Reduce下的实现原理和流程进行了说明。2.对链接分析排序算法HITS进行了分析研究,并通过引入超链接关系联合引用度和超链接文本联合相似度,提出了基于链接价值的HITS改进算法—HVHITS算法。通过结合网页信任度和蚁群算法的思想提出了基于反馈的HITS改进策略,并与HVHITS算法结合提出了—FHVHITS算法。3.在对索引、排序等相关理论研究的基础上,基于Hadoop完成了分布式搜索引擎系统的设计与实现。其中,索引及检索模块中对基于Map Reduce和DHT的混合索引策略进行了并行化实现,在链接分析排序模块中通过Map Reduce对FHVHITS算法进行并行化实现。4.在本文最后选取相应的主题及评估方法对分布式搜索引擎系统和HITS改进算法的性能进行了测试和评估。
[Abstract]:Today, the Internet has developed into the era of popular participation, people use the network more and more diverse, the more convenient to publish information, which makes the network full of massive data. How to index and retrieve these massive data is the focus of search engine. Obviously, the centralized index can no longer meet the requirements of the current big data environment. For this reason, people combine distributed technology with index technology to realize distributed index and retrieval. What kind of index strategy is used in distributed index plays a key role in the efficiency of index and retrieval, so the index strategy has important research value. Search engines get all kinds of data, and when users search, they retrieve a lot of result pages. How to sort web pages in search results according to their importance, so that the most important pages are ranked to the front of the output results, it is a very meaningful research work to save the time for users to find information. Therefore, from the above point of view, the distributed index strategy and link sorting algorithm are studied in this paper. I did the following work: 1. The related theories of distributed search engine are studied, especially the distributed index strategy, including local index strategy and global index strategy, and a hybrid index strategy based on Map reduce and DHT is proposed. At the same time, the realization principle and flow chart of Map reduce are explained. 2. 2. By introducing the hyperlink relation co-reference degree and the hyperlink text association similarity, an improved HVHITS algorithm based on link value is proposed. Based on the idea of web page trust and ant colony algorithm, the improved strategy of hits based on feedback is put forward, and the algorithm of -FHVHITS. 3 is proposed in combination with HVHITS algorithm. The design and implementation of distributed search engine system based on Hadoop is completed on the basis of the research of index and sorting theory. In the index and retrieval module, the hybrid index strategy based on Map reduce and DHT is parallelized, and the FHVHITS algorithm is parallelized by Map reduce in the link analysis and sorting module. At the end of this paper, the performance of distributed search engine system and hits improved algorithm is tested and evaluated.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【参考文献】