云存储中数据编码优化和重复数据删除技术研究

发布时间：2018-04-25 02:40

本文选题：云存储 + 纠删码　；参考：《东北大学》2013年硕士论文

【摘要】：随着云计算时代的到来,各种应用程序需要分析计算的数据所占用的空问越来越大,一度由GB到TB甚至发展到EB数量级。随着数据量的增大,需要的存储空间也变得越来越大,相应的能源消耗也越来越大。在云存储环境下,研究如何利用纠删码编码技术和重复数据删除技术有效的提高存储空间利用率是有现实意义的。本文对现有的云存储技术进行了分析,现有研究表明以纠删码为容灾策略的数据存储技术相比多副本存储技术能够节省更多的存储空间和网络带宽；在云计算平台Hadoop中,大量数据之间存在重复数据,而重复数据删除技术能发现这些重复的数据,避免重复数据的存储,提高存储空间利用率。柯西里德-所罗门编码是一种主流的纠删码编码方式,为了优化云存储中柯西里德-所罗门编码效率,并向云储存系统中加入重复数据删除功能,本文做了以下几方面的工作：(1)提出了一种优化柯西里德-所罗门编码效率的选择框架。任何利用柯西里德-所罗门编码的数据存储系统都可以利用该选择框架产生高效的数据编码调度,以用于提高编码效率。(2)将上述选择框架产生的优化调度方案集成到了Hadoop中去,优化了云存储环境下的柯西里德-所罗门编码的编码效率。(3)在Hadoop平台的每个数据节点实现了重复数据删除功能,提高了每个数据节点的存储空间利用率。在Hadoop云存储实验平台上实现了柯西里德-所罗门编码效率的优化和重复数据删除功能。本文提高了云存储中柯西里德-所罗门编码效率,提高了存储空间利用率。
[Abstract]:With the advent of cloud computing, applications need to analyze the calculation of the data occupied more and more empty question, once from GB to TB and even developed to EB order of magnitude. As the amount of data increases, the storage space needed becomes larger and larger, and the corresponding energy consumption becomes larger and larger. In the cloud storage environment, it is of practical significance to study how to use erasure coding technology and repeated data deletion technology to effectively improve the utilization of storage space. In this paper, the existing cloud storage technology is analyzed. The existing research shows that the data storage technology with erasure code as disaster recovery strategy can save more storage space and network bandwidth than multi-copy storage technology. There are repeated data among a large number of data, and repeated data deletion technology can find these duplicated data, avoid the storage of duplicate data, and improve the utilization of storage space. Caucheld-Solomon coding is a mainstream erasure coding method. In order to optimize the efficiency of Cauchy Reed-Solomon coding in cloud storage, and add duplicate data deletion function to cloud storage system, In this paper, the following work is done: 1) A selection framework is proposed to optimize the efficiency of Caucheld-Solomon coding. Any data storage system using Caucheld-Solomon coding can use the selection framework to generate efficient data coding scheduling, which can be used to improve coding efficiency. (2) the optimal scheduling scheme generated by the above selection framework is integrated into Hadoop. The coding efficiency of Caucheld-Solomon coding in cloud storage environment is optimized. The efficiency of repeated data deletion is realized at each data node in Hadoop platform and the storage space utilization ratio of each data node is improved. The optimization of Caucheld-Solomon coding efficiency and the repeated data deletion are realized on the Hadoop cloud storage experimental platform. In this paper, the efficiency of Cauchy-Solomon coding in cloud storage is improved, and the utilization of storage space is improved.
【学位授予单位】：东北大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】