重复数据删除技术在图片文件系统中的应用

发布时间：2018-01-21 05:27

本文关键词： 重复数据删除存储备份 JPEG　出处：《成都理工大学》2013年硕士论文　论文类型：学位论文

【摘要】：论文基于作者在“磁盘上实现物理磁带库(PTL)功能的虚拟磁带库(VTL)”项目开发中的工作,结合了作者对存储领域的一些思考,对移动终端数字信息结构和重复数据删除技术的发展方向以及带重复数据删除技术的存储系统的功能设计、开发方面等进行了较为系统的探索,实现了一款易部署、廉价的基于linux服务器的带重复数据删除技术的图片存储系统。重复数据删除已经是数据存储备份领域的必备技术。随着信息数字化的不断推进,人们需要存储、传输的数据量会越来越多,数据信息结构多样化化也会越来越明显。3G甚至4G无线网络的普及进一步促进了移动终端信息结构的调整,人们需要从移动终端获取更丰富的数据信息的趋势也越来越明显,这种丰富的数据信息需求刺激着信息主体由文字转变为图片,甚至进一步转变为视频。论文首先介绍了重复数据删除涉及的关键技术点及其在实际应用中所面临的问题,然后提出了超大量图片存储所面对的不同于一般数据存储的特殊需求,围绕着重复数据删除技术应用于图片存储的问题进行了深入的探讨,其中以建立一个开放的、易实现的、广泛兼容的图片存储系统逻辑结构为重点研究内容,以占服务器市场90%以上的linux操作系统为图片存储系统的开发和部署平台。利用linux操作系统的虚拟文件系统(VFS)的广泛支持,以及VFS对网络文件系统的支持兼容,开发实现了一款基于VFS的图片存储系统。针对图片文件的特性,选用CDC算法进行图片文件的数据分块,并且结合linux下的磁盘文件系统ext3以及磁盘物理特性对CDC算法的预设参数进行合理的设置,以期得到适用于图片存储的文件分块算法。针对超大量图片文件的访问问题,提出了利用文件名nash值加快文件搜索的方法。论文以source insight为项目代码管理工具,以gcc为代码编译工具,结合了图片文件存储访问操作主要以文件而不是数据块为单位的特性,直接利用linux的VFS文件系统的文件操作的概念实现了一款极易部署的图片重删存储系统。该存储系统是以动态链接库的形式存在,提供了进行图片存储与访问所需的基本操作接口,并且可以容易的根据实际应用的需要扩展出其他的功能接口。
[Abstract]:Based on the author's work in the project of "Virtual tape library VTL for realizing the function of physical tape library PTL on disk", this paper combines the author's thinking on the field of storage. The development direction of digital information structure and repetitive data deletion technology of mobile terminal and the function design and development of storage system with repetitive data deletion technology are systematically explored. An easy-to-deploy, low-cost image storage system with duplicate data deletion technology based on linux server is implemented. Repeated data deletion is an essential technology in the field of data storage and backup. With the development of information digitization, people need to store and transmit more and more data. The diversification of data information structure will also become more and more obvious. The popularity of 3G and even 4G wireless network will further promote the adjustment of mobile terminal information structure. The trend that people need to obtain more abundant data information from mobile terminal is also more and more obvious. This kind of rich data information demand stimulates the information subject to change from the text to the picture, and even to the video. This paper first introduces the key technologies involved in repeated data deletion and the problems it faces in practical applications, and then puts forward the special requirements of super-mass picture storage which are different from general data storage. In this paper, the application of repeated data deletion in picture storage is deeply discussed, and the emphasis is on the establishment of an open, easy to implement and widely compatible logical structure of picture storage system. The linux operating system, which accounts for more than 90% of the server market, is used as the development and deployment platform of the picture storage system. The virtual file system using the linux operating system is widely supported. As well as the compatibility of VFS to the network file system, a picture storage system based on VFS is developed and implemented. According to the characteristics of the picture file, the CDC algorithm is selected to divide the data of the picture file. And combining the disk file system ext3 under linux and the physical characteristics of the disk, the preset parameters of the CDC algorithm are set reasonably. Aiming at the problem of accessing a large number of picture files, this paper proposes a method to speed up file search by using the nash value of file name. This paper takes source insight as the project code management tool and gcc as the code compiling tool. Combines the image file storage access operation mainly in the file rather than the data block as the unit characteristic. This paper directly utilizes the concept of file operation of VFS file system of linux to realize a picture redelete storage system which is easy to deploy. The storage system is in the form of dynamic link library. Provides the basic operation interface for picture storage and access, and can easily expand other functional interfaces according to the needs of practical applications.
【学位授予单位】：成都理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】