基于邻接特征的网页主题传播算法

发布时间：2018-11-28 13:38

【摘要】：本文围绕邻接特征的选择、表示和使用进行相关研究,提出了基于邻接特征的网页主题传播算法。该算法基于网页超链接构建的有向图模型,针对网页的主题局部性特点,实现了邻接网页的主题传播,并最终得到各网页的主题概率分布。本文的主要工作如下:1)利用网络爬虫对互联网数据进行采集,经网页正文提取、网页去重、超链接分析等数据预处理后,构建基于网页超链接的有向图;2)利用潜在狄利克雷分布计算超链接有向图中网页节点的初始主题概率分布。所得主题概率分布作为网页节点的特征表示,一方面可以起到降维作用;另一方面可以用来衡量网页间的语义相似度;3)针对邻接特征的表示和引入问题,本文提出了虚拟节点和虚拟链接的概念。虚拟节点由目标页面的双亲页面转化而来,并包含所有指向目标网页超链接的扩展锚文本信息。虚拟链接可以将目标网页的其他邻接特征以人为链接的形式引入到模型当中。通过虚拟节点和虚拟连接,本文实现了网页邻接特征的有效表示和引入;4)针对网页主题局部性特点,本文提出了基于邻接特征的网页主题传播算法,该算法基于网页超链接构建的有向图模型,实现了邻接网页的主题传播,并最终生成各网页的主题概率分布。实验表明,在困惑度指标下,相较于网页的初始主题概率分布,基于邻接特征的网页主题传播算法对于网页主题概率分布的计算有明显提升。
[Abstract]:This paper focuses on the selection, representation and use of adjacent features, and proposes a topic propagation algorithm based on adjacent features. The algorithm is based on the directed graph model constructed by hyperlinks of web pages. According to the local characteristics of web pages, the topic propagation of adjacent web pages is realized, and the topic probability distribution of each web page is obtained. The main work of this paper is as follows: 1) the web crawler is used to collect the Internet data. After the data preprocessing, such as page text extraction, web page removal and hyperlink analysis, a directed graph based on the web page hyperlink is constructed. 2) using the potential Delikley distribution to calculate the initial topic probability distribution of the webpage nodes in hyperlink digraphs. As the feature representation of web nodes, the topic probability distribution can reduce the dimension, on the other hand, it can be used to measure the semantic similarity between web pages. 3) the concept of virtual node and virtual link is proposed in this paper. The virtual node is transformed from the parent page of the target page and contains all extended anchor text information pointing to the target page hyperlink. Virtual link can introduce other adjacent features of the target web page into the model. Through virtual nodes and virtual connections, this paper realizes the effective representation and introduction of the adjacent features of web pages. 4) according to the local characteristics of web pages, this paper proposes an algorithm of topic propagation based on adjacent features. The algorithm is based on the directed graph model constructed by hyperlinks, and realizes the topic propagation of adjacent web pages. Finally, the topic probability distribution of each web page is generated. The experimental results show that, compared with the initial topic probability distribution of the web pages, the algorithm based on the adjacent features can significantly improve the calculation of the probability distribution of the page topics under the confusion index.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092;TP391.1

【相似文献】