用户感知的重复数据删除算法

发布时间：2018-05-15 03:17

本文选题：重复数据删除 + 云计算　；参考：《软件学报》2015年10期

【摘要】：通过大量的实验分析发现:在云桌面场景下,数据拥有者之间的工作相关度越大,则该用户之间存在重复数据的概率越大.基于该实验结果,提出了用户感知的重复数据删除算法.该算法打破了数据空间局部性特征的限制,实现了以用户为单位的更粗粒度的查重计算,可以在不影响重删率的前提下,减少5~10倍常驻内存指纹的数量,并可将每次查重计算的指纹检索范围控制在一个常数范围内,不随数据总量的增加而线性增加,从而有效避免了因为数据总量增加而导致内存不足的问题.除此之外,该算法还能根据存储系统的负载情况自动调整重复指纹检索范围,在性能与重删率之间加以平衡,从而更好地满足主存储场景的需要.原型验证表明,该算法可以很好地解决云计算场景下海量数据的重复数据删除性能问题.与Open Dedup算法相比,当数据指纹总量超出内存可用空间时,该算法可以表现出巨大的优势,减少200%以上的读磁盘操作,响应速度提升3倍以上.
[Abstract]:Through a large number of experiments, it is found that in the cloud desktop scenario, the greater the work correlation between the data owners, the greater the probability of duplicate data among the users. Based on the experimental results, a user-aware repetitive data deletion algorithm is proposed. The algorithm breaks the limitation of the local feature of data space, and realizes a coarser granularity recalculation based on the user. It can reduce the number of resident memory fingerprints by 5 ~ 10 times without affecting the rate of redelete. The range of fingerprint retrieval can be controlled within a constant range and not linearly increased with the increase of the total amount of data, thus effectively avoiding the problem of insufficient memory caused by the increase of the total amount of data. In addition, the algorithm can automatically adjust the range of repeated fingerprint retrieval according to the load of the storage system, and balance the performance with the rate of redelete so as to better meet the needs of the main storage scene. Prototype verification shows that the algorithm can solve the problem of duplicate data deletion performance in cloud computing scenarios. Compared with the Open Dedup algorithm, when the total amount of fingerprint data exceeds the memory available space, the algorithm can show great advantages, reduce the read disk operation by more than 200%, and increase the response speed by more than three times.
【作者单位】：武汉大学计算机学院;深圳华为技术有限公司IT标准与专利部;国网湖南省电力公司信息通信公司;
【基金】：国家自然科学基金(61272454) 高等学校博士学科点专项科研基金(20130141110022)
【分类号】：TP333

【参考文献】