基于网站层次结构和主题模型LDA的网站自动摘要

发布时间：2018-04-29 20:19

本文选题：Web网页 + 网站自动摘要　；参考：《计算机工程》2017年04期

【摘要】：近年来自动摘要方面的研究大多是关于多文档和Web网页的,而对网站自动摘要的研究较少。为此,基于主题模型隐含狄利克雷分布(LDA)和网站层次结构提出一个可以自动生成网站摘要的算法。该算法可获取整个网站内的网页信息并进行整合,根据提出的句子权重公式计算句子权重,选取权重最高的句子作为网站摘要。以20个商业和学术网站作为实验对象,使用ROUGE评测标准,结果表明,与仅使用主题模型LDA获取的网站摘要相比,不带停用词的ROUGE-1和ROUGE-L提高0.32,带停用词的ROUGE-1提高0.39,ROUGE-L提高0.38。与网站首页摘要相比,不带停用词的ROUGE-1提高0.03,ROUGE-L提高0.06,带停用词的ROUGE-1提高0.08,ROUGE-L提高0.07。
[Abstract]:In recent years, the research on automatic summary is mostly about multi-document and Web pages, but the research on automatic summary of Web site is less. For this reason, an algorithm for automatically generating web abstracts is proposed based on the implicit Delikley distribution (LDAs) and the hierarchical structure of the web site. The algorithm can obtain and integrate the web page information of the whole website, calculate the sentence weight according to the proposed sentence weight formula, and select the sentence with the highest weight as the summary of the website. Using 20 commercial and academic websites as experimental subjects and using ROUGE evaluation criteria, the results show that the ROUGE-1 and ROUGE-L without deactivating words and ROUGE-1 with deactivated words are 0.32 and 0.39% higher than those obtained by using LDA only, and 0.39% of ROUGE-1 with deactivated words are increased by 0.38. Compared with the first page summary of the website, the ROUGE-1 without deactivation words increased 0.03% ROUGE-L increased 0.06%, the ROUGE-1 with deactivated words increased 0.08% ROUGE-L increased 0.07%.
【作者单位】：华东师范大学计算机科学技术系;
【基金】：国家科技支撑计划项目(2015BAH01F02) 上海张江国家自主创新示范区专项发展资金计划项目(201411-JA-B108-002)
【分类号】：TP391.1;TP393.092

【相似文献】