基于Docker技术的全文搜索引擎的研究与应用

发布时间：2018-05-29 05:25

本文选题：Hadoop + Map/Reduce　；参考：《南京邮电大学》2017年硕士论文

【摘要】：随着计算机世界第三次革命浪潮的兴起。在这次浪潮中,云计算与大数据大量的应用,使得数据的处理已经跃升至TB乃至PB级,并同时针对这些数据进行更快捷、更高效的处理。因此,在云计算概念上衍生而来的各种大数据处理方法与技术,业已成为此次浪潮中主流[20]。而Hadoop平台作为此次浪潮中应用最广泛的大数据处理平台,构建在基于虚拟化技术的Hadoop架构全文搜索引擎的上基础上,有着运行稳定、经济、便于管理、存储和计算的优势。本文在全文搜索引擎的搭建方面,首先通过分析和总结当前几种分布式搜索引擎的优缺点后,提出基于Hadoop平台的分布式搜索引擎,然后分析传统服务器部署的局限性并比较传统的虚拟化技术与Docker容器技术在处理性能方面的优劣,从而使用Docker容器作为Hadoop平台底层架构来搭建Hadoop平台,以便优化Hadoop平台的性能。接着,对分布式搜索引擎的爬行、索引、查询三个子系统进行研究,并应用Map/Reduce的并行算法思想,使Map函数封装数据计算任务、Reduce函数封装数据合并任务。此外,系统在全文检索方面使用了基于倒排文档的技术并结合TF-IDF(Term frequency inverse document frequency)和PageRank算法进行相关度计算,优化检索方法。同时,经过底层Docker容器可以更方便的进行搜索引擎的部署和移植。基于以上研究,本文先通过对比实验,验证了与传统虚拟技术相比,Docker在读写性能方面的优势。接着,设计与优化了Hadoop在Docker容器集群的部署方案。基于以上两点,设计与构建了一个基于Docker技术的Hadoop架构的全文搜索引擎系统,并对系统的性能、可靠性、可扩展性进行测试。通过对获取的实验数据进行分析,验证了基于Docker技术的Hadoop架构的全文搜索引擎的合理性与正确性。
[Abstract]:With the rise of the third wave of revolution in the computer world. In this wave, cloud computing and big data applications make the data processing has jumped to TB and even PB level, and at the same time for these data faster and more efficient processing. Therefore, various big data processing methods and technologies derived from cloud computing concepts have become the mainstream of this wave [20]. The Hadoop platform, as the most widely used big data processing platform in this wave, is built on the basis of the full-text search engine of Hadoop architecture based on virtualization technology. It has the advantages of stable operation, economy, easy management, storage and computing. In the construction of full-text search engine, first of all, by analyzing and summarizing the advantages and disadvantages of several kinds of distributed search engines, a distributed search engine based on Hadoop platform is proposed. Then the limitations of traditional server deployment are analyzed and the advantages and disadvantages of traditional virtualization technology and Docker container technology in handling performance are compared so that the Docker container is used as the Hadoop platform infrastructure to build Hadoop platform in order to optimize the performance of Hadoop platform. Then, the crawling, indexing and querying subsystems of distributed search engine are studied, and the parallel algorithm of Map/Reduce is applied to make Map function encapsulate data computing task and reduce function encapsulate data merge task. In addition, in the aspect of full-text retrieval, the technology based on inverted documents is used to optimize the retrieval method by combining TF-IDF(Term frequency inverse document frequency) and PageRank algorithm to calculate the correlation degree. At the same time, through the underlying Docker container can be more convenient to deploy and transplant search engines. Based on the above research, this paper verifies the advantages of Docker in reading and writing performance compared with traditional virtual technology through comparative experiments. Then, the deployment scheme of Hadoop in Docker container cluster is designed and optimized. Based on the above two points, a full-text search engine system with Hadoop architecture based on Docker technology is designed and constructed, and the performance, reliability and extensibility of the system are tested. Through the analysis of the obtained experimental data, the rationality and correctness of the full-text search engine based on Hadoop architecture based on Docker technology are verified.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【参考文献】