基于Hadoop的分布式Web文本检索系统的研究与开发

发布时间：2018-03-14 01:49

本文选题：云计算　切入点：维吾尔语文本检索　出处：《新疆大学》2013年硕士论文　论文类型：学位论文

【摘要】：本文主要是Hadoop云平台上的Nutch分布式文本检索系统的研究和开发，，检索系统运行在由多台PC机组成的集群上，系统的数据处理采用MapReduce分布式编程模型，数据存储采用分布式文件系统HDFS，系统各模块之间通过HDFS来进行数据交互，数据爬虫模块将抓取后的原始数据存入HDFS的抓取数据库中，索引模块从HDFS上读取抓取的原始数据，并使用倒排索引的方法将建好的索引存储在HDFS上的索引数据库中，检索模块则从HDFS中查找索引，向用户返回结果，整个系统通过数据抓取、索引和检索等模块的协同工作来满足用户的检索请求。在确保Nutch在Hadoop上的正常运行后进行二次开发，即解决维吾尔文的在线输入及切换、网页布局、书写方向、字库嵌入等问题。建立了一个支持维吾尔文字的基于Hadoop的Web分布式文本检索系统。为了保障系统的可靠性，系统在设计上避免了单点故障，即在单个服务器瘫痪的情况下整个系统仍然可以正常工作，同时系统还采用了不同等级的备份措施，对所有数据都进行了不同程度的备份，提高了数据的安全性。以下是在设计和研究的过程中所做的具体工作： 1.分析了Hadoop开源云计算平台和Nutch搜索引擎的相关技术、特点以及工作原理； 2.搭建具有三个节点的Hadoop平台； 3.在Hadoop平台上安装和配置Nutch开源搜索引擎； 4.根据维吾尔文的文字特征以及其处理的相关关键技术的基础上对Nutch进行了二次开发。
[Abstract]:This paper is mainly about the research and development of the Nutch distributed text retrieval system on the Hadoop cloud platform. The retrieval system runs on a cluster composed of multiple PCs. The data processing of the system adopts the MapReduce distributed programming model. The distributed file system (HDFS) is used for data storage, and the modules of the system interact with each other through HDFS. The data crawler module stores the captured raw data into the grabbing database of HDFS, and the index module reads the fetched raw data from HDFS. And the inverted index method is used to store the built index in the index database on HDFS. The retrieval module looks up the index from the HDFS and returns the result to the user. The whole system grabs the index through the data. After ensuring the normal operation of Nutch on Hadoop, the secondary development is carried out, which is to solve the online input and switch of Uygur language, the layout of web page, the direction of writing, and so on. This paper presents a Web distributed text retrieval system based on Hadoop, which supports Uygur characters. In order to ensure the reliability of the system, the system is designed to avoid a single point of failure. In the case of single server paralysis, the whole system can still work normally. At the same time, the system also uses different levels of backup measures, all the data are backed up to varying degrees, improving the security of the data. The following are specific work done in the design and research process:. 1. The related technologies, characteristics and working principles of Hadoop open source cloud computing platform and Nutch search engine are analyzed. 2. Build a three-node Hadoop platform; 3.install and configure Nutch open source search engine on Hadoop platform; 4. Based on the characters of Uygur language and the key technologies of Uygur language processing, the second development of Nutch is carried out.
【学位授予单位】：新疆大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【相似文献】