藏文文本复制检测技术研究

发布时间：2018-08-09 07:32

【摘要】：文本是互联网信息资源的一种主要存在形式。随着互联网的不断发展和网络数字资源的日益丰富,给人们提供了资源共享和信息交流的便利平台。已经成为人们信息获取的重要来源,同时为广大科研工作者和师生提供了便捷的学术交流机会。一个文本经过增添删改字词或改变说法重述之后便可以形成新的文本,这种行为称为文本的复制或抄袭。文本复制检测技术就是防止这种行为、保护文本知识产权、端正学术风气和提高信息检索效率的重要手段。目前,中英文文本复制检测技术较成熟。但由于藏文与中英文语言天然存在差异,诸多中英文自然语言的复制检测技术并不能完全适用于藏文,也无法用它们来检测藏文文本的复制率。这一空白导致了很多民族高校和藏学研究者出现了论文质量低、学术气氛差和学术创新难以提高等现象。那么,如何针对这种现象设计并实现藏文文本的复制率检测系统是本课题研究的重点。经过分析中英文复制检测结果,发现一般抄袭者所抄袭的最小单元不会小于句子这个粒度。因为句子是文本内容中具有完整语义的基本文本单元。因此,本文是基于藏文句子级别的复制检测方法,利用空间向量的余弦相似度算法计算藏语句子的相似度。该算法的关键是选取特征向量,用特征向量生成向量空间模型,最后计算余弦相似度。文中对文本复制检测技术进行了研究。根据文本复制检测的基本步骤对藏文文本进行预处理、文本分块、特征提取、句子相似度计算,最后用句子相似度来度量整篇藏文文本的抄袭率。在藏文文本预处理时,考虑了编码的统一性和可存储性分别对藏文文本的编码和藏文字符编码进行了研究,将其统一转换成Unicode编码。在藏文文本分块时,采用了藏文句子边界识别方法,把藏文文本按句子为粒度进行分块处理。同时建立句子与文档的倒排索引表,减少重复句子的两两比较和定位句子的位置信息。在藏文文本的特征提取时,采用了藏文自动分词方法,用TF-IDF计算每个词的频率,构建词频向量集。其次,计算待检测文本的每个文本块与库文本的文本块之间的相似度来度量整片文本的复制率。最后,用待检测文本进行测试,对测试结果进行了比较和分析,用查准率和查全率两个性能指标评价藏文文本复制检测技术。
[Abstract]:Text is one of the main forms of Internet information resources. With the continuous development of the Internet and the increasingly rich digital resources of the network, it provides a convenient platform for people to share resources and exchange information. It has become an important source for people to obtain information, and provides convenient opportunities for scientific research workers and teachers and students to communicate with each other. A new text can be formed after a text has been added, deleted, or restated, which is called a copy or plagiarism of the text. Text copy detection technology is an important means to prevent this kind of behavior, to protect the intellectual property rights of text, to correct the academic atmosphere and to improve the efficiency of information retrieval. At present, Chinese and English text copy detection technology is more mature. However, due to the natural differences between Tibetan and Chinese, many natural language replication and detection techniques can not be fully applicable to Tibetan, nor can they be used to detect the copy rate of Tibetan text. This gap has led to the low quality of papers, poor academic atmosphere and difficult to improve academic innovation in many ethnic colleges and universities and Tibetology researchers. So, how to design and implement the copy detection system of Tibetan text is the focus of this research. It is found that the minimum unit of plagiarism is not smaller than the grain size of sentence. Because sentence is the basic text unit with complete semantics in text content. Therefore, this paper is based on Tibetan sentence level replication detection method, using space vector cosine similarity algorithm to calculate the similarity of Tibetan sentences. The key of the algorithm is to select the feature vector, generate the vector space model with the feature vector, and calculate the cosine similarity finally. In this paper, the text copy detection technology is studied. According to the basic steps of text copy detection, the Tibetan text is preprocessed, partitioned, feature extracted, sentence similarity calculated, and the plagiarism rate of the whole Tibetan text is measured by sentence similarity. In the preprocessing of Tibetan text, the unity and storability of encoding are considered, and the encoding of Tibetan text and Tibetan character coding are studied, respectively, and the unified conversion to Unicode coding is carried out. When the Tibetan text is divided into blocks, the Tibetan sentence boundary recognition method is adopted, and the Tibetan text is processed in blocks according to the grain size of the sentences. At the same time, the inverted index table of sentence and document is established to reduce the pairwise comparison of repeated sentences and locate the position information of sentences. In the feature extraction of Tibetan text, the Tibetan automatic word segmentation method is adopted, the frequency of each word is calculated by TF-IDF, and the word frequency vector set is constructed. Secondly, the similarity between each block of text to be detected and the block of library text is calculated to measure the copy rate of the whole piece of text. Finally, the test results are compared and analyzed with the text to be tested, and two performance indexes, precision and recall, are used to evaluate the copy detection technology of Tibetan text.
【学位授予单位】：青海民族大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TP391.1

【相似文献】