基于Web的动态评论抽取技术研究

发布时间：2018-02-20 21:09

本文关键词： 信息抽取动态页面 Chrome LFSU DOM　出处：《沈阳航空航天大学》2014年硕士论文　论文类型：学位论文

【摘要】：Web2.0时代的到来推动互联网由过去的信息发布平台转变为如今的信息交互平台，在这一平台上人们可以就自己感兴趣的话题发表意见、参与讨论并形成舆论效应，其中不乏恶意利用网络舆论者，因此舆情分析工作也越来越受到重视，而Web信息抽取则是舆情分析的基础工作。 Web信息抽取是从无结构或半结构的网页中抽取特定信息的结构化描述。本文介绍了web信息抽取技术现状，针对现有技术对网页结构敏感、动态多级评论抽取研究较少等问题设计了一种半自动的信息抽取系统，该系统主要分为信息源获取与评论抽取两大模块。信息源获取模块是基于Chrome插件技术、利用浏览器API与消息传递机制开发的页面抓取工具，实现了动态页面完整内容的自动获取。评论抽取模块基于动态页面的视觉、结构、语义特征提出了LFSU概念，，利用其定位性质进行不同评论组织模型下的评论区域识别，并给出了单级评论与多级评论的抽取方法。该信息抽取方法利用少数DOM树信息，并且不涉及复杂结构比对与聚类分析，算法效率高。通过实际环境下覆盖性实验结果分析发现，该信息抽取方法满足了博客舆情数据实际分析需求，对于评论数量大于1的页面有很好的抽取效果。其查全率、查准率和F值均达到92%以上。
[Abstract]:The advent of the Web2.0 era has transformed the Internet from a former information publishing platform to a modern information exchange platform, where people can express their views on topics of interest to themselves, participate in discussions and form a public opinion effect. Many of them use public opinion maliciously, so more and more attention is paid to the analysis of public opinion, and Web information extraction is the basic work of public opinion analysis. Web information extraction is a structured description of extracting specific information from unstructured or semi-structured web pages. This paper introduces the present situation of web information extraction technology. A semi-automatic information extraction system is designed for dynamic multi-level comment extraction, which is mainly divided into two modules: information source extraction and comment extraction. The information source extraction module is based on Chrome plug-in technology. A page capture tool developed by browser API and message passing mechanism is used to realize the automatic acquisition of the complete content of dynamic pages. The comment extraction module is based on the visual structure and semantic features of dynamic pages and puts forward the concept of LFSU. Using its localization property to identify comment regions under different comment organization models, a method of extracting single and multilevel comments is presented. This information extraction method uses a few DOM tree information. The algorithm is efficient and does not involve complex structure alignment and clustering analysis. By analyzing the results of overlay experiments in actual environment, it is found that this information extraction method meets the demand of actual analysis of blog public opinion data, and has a good extraction effect for pages with a number of comments greater than 1. The precision rate and F value were above 92%.
【学位授予单位】：沈阳航空航天大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.09;TP391.1

【参考文献】