基于兴趣推送的主题爬虫研究

发布时间：2018-02-09 08:58

本文关键词： 数码产品主题相关度预测兴趣推送主题爬虫主题漂移　出处：《昆明理工大学》2012年硕士论文　论文类型：学位论文

【摘要】：Internet发展速度迅猛,互联网上的信息量越来越大,通用网络爬虫的局限性随之显现出来,用户对其查全率和查准率的需求已得不到满足,特别是对某一领域内的搜索结果不能令用户满意。针对此问题,研究人员在通用爬虫的基础上发明了主题网络爬虫,它在一定程度上改善了通用网络爬虫查准率不足的缺点。本文对Google的PageRank算法进行了详细地分析,并对其作了基于链接结构和网页内容的主题相关性改进,提出了结合网页链接和文本内容的主题爬虫算法,对搜索结果进行主题相关度排序,提高了通用网络爬虫的查准率。且针对目前的主题爬虫还存在每次需要点击许多相关网页才能找到感兴趣的内容的问题,本文提出一种基于兴趣推送的主题网络爬虫,通过用户点击行为来发现用户兴趣的兴趣推送算法,并将其与前面的主题相关性结合,使用户使用搜索引擎时不仅能在互联网巨大的信息量中快速找到行业相关网页,而且通过分析和记录用户兴趣,将用户最感兴趣的网页推送到搜索结果前面。本文以Eclipse为开发环境,开发了一套数码产品搜索引擎系统来验证改进后的算法的优越性,作者将改进后的算法应用于此系统中。通过对系统的分析和测试,本文设计和实现的基于兴趣推送的数码产品主题爬虫的方案是切实可行的,提高了搜索引擎查询的效率和准确度,且在较大程度上为用户提供了更加便利的搜索服务。
[Abstract]:With the rapid development of Internet and the increasing amount of information on the Internet, the limitations of common web crawlers appear, and users' demand for recall and precision has not been met. In particular, the search results in a certain field can not satisfy the users. In order to solve this problem, the researchers have invented the topic web crawler based on the general crawler, which to some extent improves the shortcomings of the common web crawler precision rate. In this paper, the PageRank algorithm of Google is analyzed in detail, and the topic correlation improvement based on link structure and web content is made, and a topic crawler algorithm combining web link and text content is proposed. The search results are ranked by the correlation degree of the topic, which improves the accuracy of the common web crawler, and the problem of the current topic crawler still needs to click on many relevant pages each time to find the content of interest. In this paper, a topic crawler based on interest push is proposed, which can discover the interest push algorithm of user interest by user click behavior, and combine it with the previous topic correlation. When users use search engines, they can not only quickly find relevant pages in the huge amount of information on the Internet, but also push the most interesting web pages to the front of search results by analyzing and recording users' interests. In this paper, a set of digital product search engine system is developed to verify the superiority of the improved algorithm based on Eclipse. The author applies the improved algorithm to the system. The scheme designed and implemented in this paper is feasible, improves the efficiency and accuracy of search engine query, and provides a more convenient search service for users.
【学位授予单位】：昆明理工大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.3

【参考文献】