基于哈希学习的高效文本拷贝检测研究

发布时间：2018-03-20 00:31

本文选题：哈希学习　切入点：拷贝检测　出处：《复旦大学》2013年硕士论文　论文类型：学位论文

【摘要】：在信息化不断深入、互联网越来越普及的今天,各种形式的文本数据正以惊人的速度增长,由此带来的数据拷贝问题也变得越来越严重。对于企业或组织来说,大量的重复数据导致存储和检索的效率降低；对于互联网网站来说,大量的抄袭严重影响数据生产者的权益和积极性,不利于整个互联网的良性发展；同时,数据拷贝对于搜索引擎的效果也造成了一定的负面影响。对于文本拷贝检测研究来说,该领域的研究方向主要分为两部分：1)文本表示,2)效率和可扩展性。前者主要研究如何从文本中抽取相关特征,用这些特征来代表文本,从而更好地进行拷贝检测；后者在大规模数据的背景下主要研究如何高效地检测拷贝文本。然而,在很多研究中,这两个方向并不是孤立的,前者会为后者服务,一个高效的拷贝检测方案可能需要特殊的文本表示方法。此外,对于不同的应用场景,进行拷贝检测的颗粒度也有所不同；对于颗粒度更小的拷贝检测,如句子级别的拷贝检测,它对效率和可扩展性的要求将更高。本文主要的研究内容同样包括这两方面,具体如下： 1)首先,本文提出了一个完整的拷贝检测框架,包括主要的流程以及拷贝检测算法； 2)其次,本文详细讨论了用哈希编码来表示文本的可行性,并且发现已有的哈希编码方案在准确率方面仍有着较大的提升空间：基于哈希编码空间有限,需要充分利用编码空间这一事实,本文提出了一个哈希编码学习方案,并且通过实验发现,该方案所得到的哈希编码确实是更优的,能够大幅提升检测的准确性； 3)最后,通过在GPU上实现最为耗时的关键算法,取得了超过1500倍的加速比,同时兼具良好的可扩展性。
[Abstract]:With the deepening of information technology and the increasing popularity of the Internet, various forms of text data are increasing at an alarming rate, and the problem of data copying is becoming more and more serious. A large amount of duplicate data leads to lower efficiency of storage and retrieval; for Internet websites, a large number of plagiarism seriously affects the rights and interests of data producers and enthusiasm, and is not conducive to the healthy development of the entire Internet. Data copy to search engine effect also caused certain negative influence. For the study of text copy detection, the research direction in this field is mainly divided into two parts: 1) text representation 2) efficiency and extensibility. The former mainly studies how to extract relevant features from the text and use these features to represent the text. The latter, in the context of large-scale data, mainly studies how to detect copy text efficiently. However, in many studies, these two directions are not isolated, the former serving the latter. An efficient copy detection scheme may require a special text representation method. In addition, for different application scenarios, the particle size of copy detection is different; for copy detection with smaller particle size, Such as sentence-level copy detection, it will require higher efficiency and scalability. The main contents of this paper also include these two aspects, as follows:. 1) first of all, this paper proposes a complete copy detection framework, including the main process and copy detection algorithm; 2) secondly, the feasibility of using hash coding to represent text is discussed in detail, and it is found that the existing hash coding schemes still have a large improvement space in terms of accuracy: based on the limited hash coding space, It is necessary to make full use of the fact of coding space. In this paper, we propose a hashing coding learning scheme, and through experiments, we find that the hashing coding obtained by this scheme is indeed better and can greatly improve the accuracy of detection. 3) finally, by implementing the most time-consuming key algorithm on GPU, the speedup is more than 1500 times and the scalability is good.
【学位授予单位】：复旦大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【相似文献】