基于手机产品信息的个性化搜索引擎的研究与实现

发布时间：2018-03-23 17:30

本文选题：Heritrix爬虫　切入点：Lucene搜索　出处：《湖北工业大学》2012年硕士论文

【摘要】：随着互联网的迅速发展，搜索引擎成为了人们获取信息的一个重要途径。人们不仅希望搜索引擎能够提供完整的信息资源，也对搜索引擎的服务提出了更高的要求。与传统的通用搜索引擎相比，，个性化搜索引擎因其能够为用户提供特定领域的主题搜索和个性化服务，而日益受到了研究者的重视。基于这样的背景，本论文设计了基于个性化服务的手机信息搜索引擎系统，它结合当前流行的Heritrix爬虫技术和Lucene搜索引擎框架，对本地手机信息资源搜索进行了很好地实现。本论文首先概述了搜索引擎的研究背景和工作原理，接着对Heritrix爬虫技术进行了深入的研究和进一步优化。Heritrix爬虫是目前实现Web信息抽取的主要技术之一，具有开源、可扩展的优点，同时也存在爬取信息时速度较慢的问题。本论文运用ELFHash算法对Heritrix爬虫进行了多线程优化，实现了指定网页精确的爬取,较好地提高了爬取网页的速度。本论文采用Lucene搜索技术作为个性化搜索引擎系统的搜索框架，对其索引和排序技术进行了深入的研究，并且对其排序算法进行了优化。Lucene原排序算法是根据文档中查询的关键词出现的次数，与文档中包含此关键词次数的比例进行结果排序，在检索结果页面与主题的相关度方面不够准确，不能很好地体现出页面的重要性程度。本论文采用了PageRank算法对Lucene的排序算法进行改进，通过计算用户兴趣与页面重要性的相关度，使得排序结果更加优化。在此研究基础上，本论文设计并实现了提供个性化手机信息服务的搜索引擎系统。根据用户的需求分析，本论文将系统分为页面爬取模块、页面解析模块、信息索引和信息检索模块等四个模块。其中，详细阐述了各个模块的设计思想和具体实现过程。通过对系统测试结果的分析，验证了系统功能，证明了系统的功能和性能达到了设计的要求。最后，本论文对全文进行了总结和展望。总结了本人在硕士研究生期间的工作和成果，并指出了系统存在的不足和进一步工作。
[Abstract]:With the rapid development of the Internet, search engine has become an important way for people to obtain information. Compared with the traditional universal search engine, personalized search engine can provide users with specific subject search and personalized services. Based on this background, this paper designs a mobile phone information search engine system based on personalized service, which combines the current popular Heritrix crawler technology and Lucene search engine framework. The local mobile phone information resource search is well realized. This paper first summarizes the research background and working principle of the search engine, and then makes a deep research on the Heritrix crawler technology and further optimizes the .Heritrix crawler, which is one of the main technologies to realize Web information extraction, which is open source. At the same time, there is the problem of slow crawling speed. In this paper, the ELFHash algorithm is used to optimize the Heritrix crawler, which realizes the precise crawling of specified web pages, and improves the speed of crawling web pages. In this paper, the Lucene search technology is used as the search framework of the personalized search engine system, and the indexing and sorting techniques are deeply studied. The original sort algorithm of Lucene is sorted according to the number of times the keywords appear in the document and the proportion of the number of times the keywords are included in the document. The relevance between the search result page and the topic is not accurate enough to reflect the importance of the page. In this paper, the PageRank algorithm is used to improve the sorting algorithm of Lucene. By calculating the correlation between user interest and page importance, the sorting results are optimized. On the basis of this research, this paper designs and implements the search engine system which provides the personalized mobile phone information service. According to the user's demand analysis, this paper divides the system into the page crawling module and the page analysis module. Information index and information retrieval module are four modules. Among them, the design idea and implementation process of each module are described in detail. The system function is verified by analyzing the test results of the system. It is proved that the function and performance of the system meet the requirements of the design. Finally, this paper summarizes and prospects the full text, summarizes my work and achievements during the period of master's degree, and points out the shortcomings and further work of the system.
【学位授予单位】：湖北工业大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.3

【相似文献】