虚拟磁带库中重复数据删除技术的研究与设计

发布时间：2018-06-04 06:51

本文选题：虚拟磁带库 + 重复数据删除　；参考：《西南交通大学》2013年硕士论文

【摘要】：随着全球信息化的推进,我们的社会正在步入一个信息化的社会,政府机构以及各行各业都对信息资源、信息技术以及信息产业的依赖程度越来越大,对存储空间的需求也在飞速地增长。在进行数据备份时,会备份大量相同的数据和文件,而这些相同的数据和文件占据了大量昂贵的磁盘空间。VTL (Virtual Tape Library虚拟磁带库)以其备份性能高、故障率低、可靠性高等优势被广泛用于政府机构以及各行各业的数据存储中。因此,研究可以删除VTL中重复数据的技术是势在必行的。本文首先对虚拟磁带库和重复删除技术在国内外的现状进行了分析,找出了现有重复数据删除技术中存在的问题和不足,从而确立了本文的研究出发点。研究了重复数据删除的基本原理,然后通过以下几个过程：文件数据检测分块、块哈希值计算、块哈希值查找以及哈希值保存,实现了一个基于块级的重复数据删除系统。为了弥补重复数据删除中MD5哈希算法的“哈希冲突”问题,使用了拉链法对哈希算法进行了优化,增强了数据的安全性；为了提高系统检测重复数据块的效率,对基于内容的数据检测算法进行了改进；为了提高哈希表查找的效率,使用了Bloom Filter技术对哈希表进行了优化和改进。最后,在虚拟磁带库与备份软件的环境下对系统进行了测试与分析,从测试结果得出,改进的CDC数据检测算法比FSP和SB算法具有更高的重复删除率,重复删除系统比一般的数据压缩软件具有更高的数据缩减率。
[Abstract]:With the development of global informatization, our society is stepping into an information society. Government agencies and various industries are relying more and more on information resources, information technology and information industry. The demand for storage space is also growing rapidly. When you do a data backup, you back up a lot of the same data and files, which take up a lot of expensive disk space. VTL Virtual Tape Library virtual tape library) because of its high backup performance and low failure rate. The advantages of high reliability are widely used in the data storage of government agencies and various industries. Therefore, it is imperative to study the technology that can delete duplicate data from VTL. This paper first analyzes the status quo of virtual tape library and repeat deletion technology at home and abroad, finds out the existing problems and shortcomings in the existing repeated data deletion technology, and establishes the starting point of this paper. In this paper, the basic principle of repeated data deletion is studied, and then a block level repeated data deletion system is implemented through the following processes: file data detection, block hash value calculation, block hash value searching and hash value saving. In order to make up for the "hash conflict" problem of MD5 hash algorithm in the repeated data deletion, the zipper method is used to optimize the hash algorithm to enhance the security of the data, and to improve the efficiency of the system to detect repeated data blocks. The content-based data detection algorithm is improved, and the Bloom Filter technique is used to optimize and improve the hash table in order to improve the efficiency of hash table lookup. Finally, the system is tested and analyzed under the environment of virtual tape library and backup software. From the test results, the improved CDC data detection algorithm has higher repetition deletion rate than FSP and SB algorithm. The data reduction rate of the repeated deletion system is higher than that of the normal data compression software.
【学位授予单位】：西南交通大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333.35

【参考文献】