基于统计机器翻译与主题传播的聚焦爬虫的研究与实现

发布时间：2018-01-20 01:29

本文关键词： 聚焦爬虫锚文本统计机器翻译主题传播主题相关度预测文本分类　出处：《浙江大学》2013年硕士论文　论文类型：学位论文

【摘要】：搜索引擎的发展使得互联网的使用变得更加便捷。人们通过搜索引擎,可以方便地从互联网上搜索到各行各业的相关信息。互联网的数据规模可以用海量形容,而现在的搜索引擎只收录了互联网所有资源中的一部分。爬虫是搜索引擎中一个重要部分,对于搜索引擎有着至关重要的作用。如何在有限的资源情况下爬取更多与人们感兴趣内容相关的网页,成为工业界和学术界的热门话题,聚焦爬虫也因此应运而生。本文的主要研究内容是基于统计机器翻译的锚文本主题相关度预测与基于网页主题传播的聚焦爬虫算法及其实现。一般情况下锚文本较短,在大多现有使用锚文本的聚焦爬虫中,都对锚文本进行了上下文扩展。然而对于一个与主题不相关的锚文本,在上下文扩展之后可能引入与主题相关的内容。本文将锚文本与主题之间的关系视为噪声信道中信息编码的过程,进而提出了基于统计机器翻译的锚文本主题相关度预测算法。然而仅仅使用锚文本这样的网页内容,很可能会丢掉一些虽然与主题不相关但是却包含大量与主题相关外链的网页。使用网页链接结构分析是解决上述问题的一个方法。本文在前人工作的基础上,提出了基于网页主题传播的聚焦爬虫算法,并将锚文本主题相关度预测算法融入到网页主题传播模型中,旨在提高聚焦爬虫的效果。本文最终设计并实现了一个聚焦爬虫系统原型,从互联网上爬取大量的特定主题数据,并将本文提出的算法与一些经典的聚焦爬取算法作了分析与比较。实验结果表明本文提出的算法具有更好的效果。
[Abstract]:The development of the search engine makes use of the Internet has become more convenient. People can search through the search engine, easily from the Internet to the relevant information from all walks of life. The Internet can be used to describe the massive scale of data, and now the search engine contains only a part of the Internet resources. Crawler is an important part of search engine which plays a crucial role in the search engines. How to take up more people interested in the content of the web page in the limited resources, has become a hot topic in academia and industry, focused crawler also arises at the historic moment.
The main content of this paper is the anchor text topic relevance statistical prediction based on Machine Translation and based on focused crawler algorithm and its theme "spread. In general the anchor text is short, in most of the existing crawler using the anchor text, the anchor text of context extension. However for a irrelevant the anchor text, in the context of expansion after the possibility of introducing relevant content and topic. The relationship between anchor text and theme as a process of information encoding channel noise, and then put forward the anchor text subject correlation prediction algorithm based on the statistics of Machine Translation. However, only use the anchor text page content that is likely to lose although some are not related to the topic but contains a large number of related to the theme of the chain page. Use the homepage link structure analysis is to solve the above problems a Based on previous works, we propose a focused crawler algorithm based on Web Topic propagation, and integrate anchor text topic correlation prediction algorithm into webpage topic propagation model, aiming to improve the effect of focused crawler.
The final design and a focused crawler system prototype, crawling specific topic a lot of data from the Internet, and will focus on the algorithm proposed in this paper and some classical algorithms are analysed and compared. The experimental results show that the proposed algorithm has better effect.

【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【相似文献】