基于回归检测的滑动块重复数据删除算法

发布时间：2018-03-07 12:46

本文选题：云存储　切入点：回归检测　出处：《新疆大学学报(自然科学版)》2017年03期 　论文类型：期刊论文

【摘要】：随着大数据时代的来临,重复数据在存储系统中占有很高比例,如何在保障数据可用性的前提下提高存储系统利用率问题一直是研究人员关注的热点.重复数据删除技术是一种存储系统优化技术,通过比较数据指纹确定冗余并删除,达到保障数据唯一性的目的.在重复数据分块检测过程中,无匹配指纹的块即认定为新数据存储,然而通过研究发现,未匹配块中仍然包含大量的重复数据,若能检测到未匹配块内的重复数据,则能够在一定程度上进一步提高重复数据检测率.本文提出一种基于回归检测的滑动块重复数据删除算法,对传统滑动块技术产生的未匹配数据块进行回归检测,通过对比未匹配块的结构变化进而确定数据操作类型,再根据不同的操作类型执行不同的检测算法,达到去除未匹配块内重复数据的目的.实验表明:本算法在时间开销方面比较合理,并能够有效提高重复数据检测率.
[Abstract]:With the advent of big data's time, duplicate data occupy a high proportion in the storage system. How to improve the utilization rate of storage system under the premise of ensuring the availability of data has always been the focus of attention of researchers. Repeated data deletion is a storage system optimization technology, which determines redundancy and deletes by comparing data fingerprints. In the process of repeated data block detection, the block without matching fingerprint is considered as the new data storage. However, it is found that the unmatched block still contains a large number of repeated data. If the repeated data in the unmatched block can be detected, the detection rate of repeated data can be further improved to a certain extent. In this paper, a sliding block repeat data deletion algorithm based on regression detection is proposed. The unmatched data blocks produced by the traditional sliding block technology are detected by regression. By comparing the structure changes of the unmatched blocks, the data operation types are determined, and different detection algorithms are executed according to the different operation types. The experimental results show that the proposed algorithm is reasonable in time cost and can effectively improve the detection rate of repeated data.
【作者单位】：新疆大学软件学院;新疆大学信息科学与工程学院;
【基金】：国家自然科学基金项目(61462079) 新疆维吾尔自治区高校科研计划(XJEDU2016S106)
【分类号】：TP301.6;TP333

【参考文献】