AdaTextTiling：一种基于TextTiling算法改进的自适应文本分割技术

发布时间：2018-03-23 04:21

本文选题：文本分割　切入点：TextTiling算法　出处：《华东师范大学》2017年硕士论文　论文类型：学位论文

【摘要】：伴随着计算机在日常生活中的逐渐普及,整个社会的信息科技得到迅猛地发展,互联网信息化程度也不断提高。通过互联网,人们可以非常方便的掌握世界范围内的各种信息,以及与各个地方的人进行信息互动,可以说互联网带给我们更加便捷、高效的信息化生活。而人们的频繁使用也带来了庞大的互联网数据资源,这些数据资源蕴含着极高的挖掘价值,其中文本数据就是占比非常大的互联网数据资源,而文本挖掘就是从丰富的文本数据资源中挖掘出有价值的信息。文本分割作为文本挖掘的一个重要分支,在文本信息挖掘方面也有相当重要的作用。文本分割指的是将一整篇文本看成由多个子主题文本段组成的,然后运用相关方法将一整篇文本分割成多个分割片段,每个文本片段都对应着一个子主题。文本分割的算法有很多,TextTiling算法就是比较经典的文本分割算法。本文主要是对经典的TextTiling算法进行改进,提出分割性能更好的AdaTextTil-ing算法,用于更好地对文本进行分割。本文首先是对TextTiling算法进行分析,掌握TextTiling算法的算法原理以及分析算法的不足之处,接着是进行优化,其中主要的一点是计算潜在分割点两边文本相似度时会灵活地调整文本窗口长度,因为本文认为每一个潜在分割点的最优文本窗口长度不是固定不变的。同时,本文还对TextTiling算法实现上的计算逻辑进行分析优化,提高算法计算效率,并在此基础上结合LDA主题模型进一步优化。最后通过实验,本文发现AdaTextTiling算法性能上要明显优于TextTiling算法,从而说明了 AdaTextTiling算法的有效性。
[Abstract]:With the gradual popularization of computers in daily life, the information technology of the whole society has been developed rapidly, and the information level of the Internet has been continuously improved. Through the Internet, people can easily grasp all kinds of information in the world. And information interaction with people in various places, it can be said that the Internet has brought us more convenient and efficient information life. And the frequent use of people has also brought huge Internet data resources. These data resources contain very high mining value, in which text data is a very large proportion of Internet data resources. Text mining is to mine valuable information from rich text data resources. Text segmentation is an important branch of text mining. Text segmentation means that a whole text is considered to be composed of multiple sub-topic text segments, and then the whole text is divided into multiple segmented segments by using relevant methods. Each text fragment corresponds to a subtopic. There are many text segmentation algorithms, which are the classical text segmentation algorithm. This paper mainly improves the classical TextTiling algorithm and proposes a better AdaTextTil-ing algorithm with better segmentation performance. This paper is to analyze the TextTiling algorithm, master the principle of the TextTiling algorithm and analyze the shortcomings of the algorithm, and then optimize the algorithm. The main point is that the text window length can be adjusted flexibly when calculating the text similarity between two potential segmentation points, because the optimal text window length of each potential segmentation point is not fixed. At the same time, This paper also analyzes and optimizes the computational logic in the implementation of TextTiling algorithm, improves the efficiency of the algorithm, and further optimizes the algorithm combined with the LDA topic model. Finally, through experiments, it is found that the performance of AdaTextTiling algorithm is obviously better than that of TextTiling algorithm. Thus, the effectiveness of AdaTextTiling algorithm is illustrated.
【学位授予单位】：华东师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.1

【参考文献】