云存储中集群重复数据删除系统的研究与设计

发布时间：2018-05-03 22:18

本文选题：云存储 + 重复数据删除技术　；参考：《东北大学》2014年硕士论文

【摘要】：随着大数据时代的到来,数据已呈爆炸式的增长,海量数据的存储已成为数据中心的首要问题,重复数据大量存在于信息处理与存储的各个环节中,如文件系统、邮件附件、web对象以及操作系统与应用软件中。传统的数据保护技术如周期备份、版本控制、快照等更是加速了重复数据的增长,导致网络带宽与存储资源的消耗以及存储成本的增加。为了提高存储资源利用率,降低数据管理成本,重复数据删除技术的提出与应用已经成为目前企业与数据中心的研究热点。云存储服务具有高可靠性、高通用性、高扩展性及大容量存储等特点,因此对云存储技术的研究不仅紧跟IT技术发展的趋势,而且具有较高的应用价值。在云存储环境中构建大规模、高性能、分布式重删系统,具有很大的优势与挑战。本文在云存储环境中设计了一套在线的集群重删系统架构,并在数据路由问题、索引查询优化问题等方面进行了相关研究,主要完成工作有：(1)基于开源的HDFS分布式文件系统,以集群重复数据删除技术为基础,设计并实现了具有重复数据删除功能的分布式文件系统H-Dedup。根据重复数据删除技术的特征,合理设计系统架构与软件功能模块,使得重复数据删除技术有效融入集群存储架构中。(2)设计面向重删的局部相似路由算法,基于数据相似理论,以超块粒度为路由单位,并对超块进行采样,选取少量特征指纹,以有状态路由方式进行路由匹配,减少网络带宽消耗,快速定位存储位置。使得分布式存储系统在保持较高的存储性能以及吞吐率的同时,快速定位存储节点并取得较高的数据去重率。(3)为了缓解索引查询过程中磁盘瓶颈问题,设计了基于内存的相似索引表,进行数据局部去重,以避免出现大量的磁盘随机读写现象。根据数据的局部性特点设计了全局LRU缓存,最大程度维护和利用数据的局部性,减少磁盘访问次数；为了弥补单节点内低去重率问题,设计了基于访问频率的容器热点指纹索引,提升单点内的数据去重率。
[Abstract]:With the arrival of big data era, the data has been explosive growth, the storage of massive data has become the primary problem of the data center, repeated data in all aspects of information processing and storage, such as file systems, E-mail attachment web object and operating system and application software. Traditional data protection techniques, such as periodic backup, version control, snapshot, etc, accelerate the growth of repeated data, resulting in the consumption of network bandwidth and storage resources as well as the increase of storage cost. In order to improve the utilization of storage resources and reduce the cost of data management, the proposed and applied repeated data deletion technology has become the research hotspot of enterprises and data centers. Cloud storage service has the characteristics of high reliability, high versatility, high expansibility and large capacity storage, so the research of cloud storage technology not only follows the development trend of IT technology, but also has high application value. Building large scale, high performance and distributed redelete system in cloud storage environment has great advantages and challenges. In this paper, an online cluster redelete system architecture is designed in cloud storage environment, and related research on data routing problem, index query optimization problem and so on is carried out. The main work is: 1) Open source HDFS distributed file system. Based on the technology of repeated data deletion in cluster, a distributed file system H-Dedup. which has the function of deleting repeated data is designed and implemented. According to the characteristics of repeated data deletion technology, the system architecture and software function module are designed reasonably, so that the duplicate data deletion technology can be effectively integrated into the cluster storage architecture. (2) the local similar routing algorithm for redelete is designed, which is based on the data similarity theory. The super-block granularity is taken as the routing unit and the super-block is sampled and a small number of characteristic fingerprints are selected to match the route in a stateful routing mode. The bandwidth consumption of the network is reduced and the storage location is quickly located. In order to alleviate the disk bottleneck problem in the process of index query, distributed storage systems maintain high storage performance and throughput, and quickly locate storage nodes and obtain high data removal rate. A similar index table based on memory is designed to avoid random reading and writing of a large number of disks. According to the local characteristic of the data, the global LRU cache is designed to maintain and utilize the locality of the data to the maximum extent, to reduce the number of disk access, in order to make up for the problem of low weight removal rate in a single node. A container hot spot fingerprint index based on access frequency is designed to improve the data removal rate in a single point.
【学位授予单位】：东北大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP333

【相似文献】