基于马尔科夫模型词序因子的文本相似度研究

发布时间：2018-08-14 09:45

【摘要】：在中文信息处理的应用中，文本相似度计算是最底层的工作，在所有的文字处理系统中都要用到文本相似度，使用非常广泛，比如搜索引擎、文本查重、信息检索等领域，一个文本相似度算法的好坏，主要体现在正确率和运行效率上，结果要尽可能的接近人工的方法，运行时间要尽可能的少。目前用的方法主要是基于关键词匹配的方法，很多算法的文本的特征项的抽取都是基于空间向量模型的（VSM），查询速度快，但是该类方法有明显不足之处，很多对文本相似度的计算带来的影响的因素没考虑在内，比如同义词、多义词、词序等因素。本文提出了一种基于马尔科夫模型词序因子的中文文本相似度算法，参考了很多国内外相关文献，对当前文本相似度计算方法的情况做了进一步研究，提出了新的文本相似度计算方法——采用《知网》知识结构进行相似度计算的方法，把多义词和同义词等因素考虑在内，然后结合马尔科夫模型，，把词序因素结合进去，即把语义相似度应用到马尔科夫模型当中去，综合考虑了词频、同义词、多义词、词序等因素。从传统的空间向量模型和语义相似度计算方法的对比结果可以看出，该算法具有较高的正确率，具有一定的实用性。最后，利用几个较小的文本文档对该算法进行测试，对几种不同的相似度计算方法进行测试对比，说明基于马尔科夫的相似度方法，正确率和可行性方面比传统几种计算方法有一定的提高。通过在准确度方面对本课题的研究成果进行测试上的比对。为文本相似度算法的发展提供一种新的思路。
[Abstract]:In the application of Chinese information processing, text similarity calculation is the lowest work. It is widely used in all word processing systems, such as search engine, text search, information retrieval and so on. A text similarity algorithm, mainly reflected in the accuracy and efficiency, the results should be as close as possible to the artificial method, running time should be as little as possible. At present, the methods used are mainly based on keyword matching. The extraction of text feature items in many algorithms is based on (VSM), query speed based on spatial vector model, but this kind of method has obvious shortcomings. Many factors, such as synonyms, polysemous words, word order and so on, are not taken into account. In this paper, we propose a Chinese text similarity algorithm based on word order factor of Markov model. Referring to a lot of relevant literatures at home and abroad, this paper makes further research on the current text similarity calculation method. In this paper, a new method of text similarity calculation is proposed, which uses knowledge structure to calculate similarity, including polysemous words and synonyms, and then combines word order factors with Markov model. The semantic similarity is applied to Markov model, and the factors such as word frequency, synonym, polysemy, word order and so on are considered. From the comparison of the traditional space vector model and the semantic similarity calculation method, it can be seen that the algorithm has higher accuracy and practicability. Finally, several smaller text documents are used to test the algorithm, and several different similarity calculation methods are tested and compared to illustrate the similarity method based on Markov. The accuracy and feasibility are improved compared with the traditional calculation methods. Through the accuracy of the research results of the subject to test the comparison. It provides a new idea for the development of text similarity algorithm.
【学位授予单位】：湖北工业大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP391.1

【相似文献】