分布式垂直搜索引擎的研究与设计

发布时间：2017-12-31 08:24

本文关键词：分布式垂直搜索引擎的研究与设计　出处：《电子科技大学》2016年硕士论文　论文类型：学位论文

更多相关文章： 海量数据 爬虫索引 Hadoop Noaql Hbase

【摘要】：面对如今互联网的海量数据,如何快速准确的获取自己想要的信息成为一个亟待解决的问题。百度, Google等全网搜索引擎通过全网爬虫获取网页信息,根据用户不同查询需求返回结果以满足用户的广度查询需求。而对于某些垂直网站的深度查询,需了解具体网站结构,才能对其进行特殊解析以获取其中数据,这方面全网爬虫往往“无能为力”。所以针对客户需求,高效获取垂直网站里的数据信息的研究就变得很有必要。为解决上述问题,本文设计并实现了一个分布式垂直搜索引擎,以满足用户对垂直网站信息的深度需求,并对该搜索引擎中建立索引和检索过程的性能进行了详细的测试。并对检索中的中文分词算法进行了研究和优化,中文分词算法的优劣,是影响一个搜索引擎查询效率的最重要因素之一。对搜索引擎进行功能测试表明,本系统能够每天定时通过基于Hadoop的分布式多线程爬虫采集数据,并通过redis这一基于内存的数据库存储需要解析的URL数据,然后存储详细商品数据到Hbase数据库中。当数据存储到Hbase后,SolrCloud集群对Hbase中存储的商品数据的指定属性建立索引,用户请求查询时,SolrCloud通过分布式检索返回结果。此外,对于搜索出的商品提供了商品比较及商品来源追踪,所有结点上的爬虫进程均能使用zookeeper进行监控,提高系统的安全性。对索引检索模块的性能测试可以得出不同分片的SolrCloud集群对索引检索性能的影响,进而能够根据不同的业务应用场景确定其分片数。此外,对中文分词算法的性能测试表明,改进后中文分词算法在准确率和召回率上都有一定提高。
[Abstract]:Facing the massive data of Internet nowadays, how to get the information we want quickly and accurately becomes a problem to be solved urgently. Baidu. The whole web search engine such as Google obtains the web page information through the whole web crawler, returns the result according to the user's different query demand to satisfy the user's breadth query demand, and the depth query for some vertical websites. Need to know specific website structure, can undertake special parse to it in order to obtain data, this respect whole web crawler is often "powerless". In order to solve the above problems, this paper designs and implements a distributed vertical search engine. In order to meet the users' demand for the depth of vertical website information, the performance of the indexing and retrieval process in the search engine is tested in detail, and the Chinese word segmentation algorithm in the search engine is studied and optimized. The advantages and disadvantages of Chinese word segmentation algorithm is one of the most important factors that affect the query efficiency of a search engine. This system can collect data every day through distributed multithread crawler based on Hadoop, and store URL data which need to be parsed through redis, which is a memory-based database. The detailed commodity data is then stored into the Hbase database. When the data is stored in the Hbase, the Solr Cloud cluster indexes the specified attributes of the commodity data stored in the Hbase. When the user requests the query, SolrCloud returns the results through distributed retrieval. In addition, it provides commodity comparison and commodity source tracking for the items searched. All crawler processes on all nodes can be monitored using zookeeper. Improve the security of the system. The performance test of index retrieval module can find out the impact of different sliced SolrCloud cluster on index retrieval performance. In addition, the performance test of Chinese word segmentation algorithm shows that the improved Chinese word segmentation algorithm has higher accuracy and recall rate.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.3

【相似文献】