词向量语义模型研究及在主题爬虫系统中的应用

发布时间：2018-08-21 09:59

【摘要】：爬虫,即使用程序自动获取网页上的内容,在现在已经很流行,是搜索引擎的重要组成部分,也是进行有监督机器学习模型训练的语料获取重要方法之一。然而,在某些特定领域的研究,普通爬虫不再能够满足特定语料获取的需求,因此带有特定主题的垂直领域爬虫已经日益被需要。主题爬虫需要在获取一个新的网页或网页链接时,通过判断在语义上是否与主题相关,来判断是否爬取该页面。本文使用词向量进行语义表示,并联合点对互信息方法,对新的网页链接进行判断,决策继续爬取该页面,还是放弃爬取该页面。具体内容如下。介绍自然语言处理技术、深度学习技术、语言模型。并详细介绍基于矩阵和基于向量的两种词向量表示方法。然后基于维基百科中文语料,使用不同的参数训练模型,得出实验结论,并选出某一组参数,进行下面章节的研究。为了解决一词多义的问题,本文引入点对互信息(PMI,Pointwise Mutual Information)。根据上下文信息,判断该词在此处的意思。并通过上一部分的结论,选出一个效果最好的词向量模型,联合PMI进行实验。PMI的词对表容量巨大,普通的电脑内存无法装载,针对该问题,本文将给出一种解决方法。把以上两部分运用于垂直领域爬虫系统。使用宽度优先搜索的方法进行抓取,当爬虫系统遇到一个新的链接时,使用上一部分得出的模型,判断该连接词与主题词的相关程度。使用“程序员”、“家具”、“护肤”三个主题,在百度百科上分别爬取若干页面,并保留中间扔掉的链接,人工判断每个网页是否与主题相关,从而得出准确率,召回率等,并与不使用相关词技术的普通爬虫对比,从而更加客观的判断本文的垂直领域爬虫的效果。本文提出了使用语义模型表示和点对互信息,联合进行网页链接是否与主题词相关的判定,从而筛选出与主题词相关的网页链接,并得出客观的实验效果。
[Abstract]:Crawlers, even though they use programs to automatically retrieve the content of web pages, are now very popular. They are an important part of search engines and one of the important methods of corpus acquisition for supervised machine learning model training. However, in some specific areas of research, common reptiles can no longer meet the requirements of specific data acquisition, so vertical domain crawlers with specific topics have been increasingly needed. A topic crawler needs to determine whether to crawl a new page or a web page by judging whether it is semantically related to the topic or not. In this paper, we use word vector for semantic representation, and combine point-pair mutual information method to judge the new web page link, and decide whether to continue crawling the page or to give up crawling the page. The details are as follows. This paper introduces natural language processing technology, deep learning technology and language model. Two word vector representation methods based on matrix and vector are introduced in detail. Then, based on the Chinese corpus of Wikipedia, different parameter training models are used to obtain the experimental conclusions, and a set of parameters is selected for the study of the following chapters. In order to solve the problem of polysemy, this paper introduces the point pair mutual information (PMI) Pointwise Mutual Information). Judge the meaning of the word here based on the context information. Based on the conclusion of the previous part, a word vector model with the best effect is selected, and the word pair of words combined with PMI is very large, and the common computer memory can not be loaded. In view of this problem, this paper will give a method to solve this problem. The above two parts are applied to the vertical reptile system. When the crawler system encounters a new link, the model obtained from the previous part is used to judge the correlation between the link and the subject word. Using the three themes of "programmer", "furniture" and "skin care", crawling several pages on Baidu Encyclopedia and keeping the links thrown away in the middle, we can manually judge whether each web page is related to the theme, so as to get the accuracy, recall rate, etc. And compared with the common crawler without using the related word technology, it is more objective to judge the effect of the vertical domain reptile in this paper. In this paper, the semantic model representation and point-pair mutual information are used to determine whether the web link is related to the subject word, so that the web link related to the theme word can be screened out, and the objective experimental effect is obtained.
【学位授予单位】：中国地质大学(北京)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【相似文献】