基于纠删码存储的数据维护关键技术研究

发布时间：2018-05-08 14:39

本文选题：分布式存储 + 纠删码　；参考：《国防科学技术大学》2013年硕士论文

【摘要】：随着当前云存储,大数据等技术的兴起,数据的可生存性日益受到重视,而数据容错技术则是保证数据生存性的主要方法。当前的数据容错技术主要通过冗余的方法实现,主要有副本方式和纠删码方式,纠删码方式在存储效率方面远远高于副本方式,但存在数据维护开销大的问题。本文对纠删码方案的数据维护开销问题进行了研究,提出其面临的两大问题:1.节点间维护通信开销过大;2.数据恢复计算开销过大。针对这两个问题,我们提出了解决的方法。首先,本文从数据读取的角度出发,提出了被动检测修复算法。该方法的主要思想是利用系统正常读带宽进行数据的检测,如果数据需要修复,则缓存正常解码后的数据于本地,通过这种方法检测过的数据在被修复时,不需要再次检测、下载和解码数据,降低了通信和修复开销。据我们所知,目前还没有类似的方法。其次,本文从系统可靠性的角度出发,提出了自适应检测算法,降低数据修复和通信开销。该方法的主要思想是根据系统可靠性的不同来调整数据维护的频率,可靠性较高的系统则维护频率较低。该方法与现存的其他方法相比,主要不同在于数据维护频率是根据系统可靠性而动态变化,在降低维护频率的同时,保证了数据可用性。再次,本文从数据分类的角度出发,提出了容忍度选择算法。该方法主要思想是根据数据访问模式的不同(实际的存储系统中,数据的访问模式存在较大的差异[64]),对不同的数据实现不同的维护频率,降低不经常访问数据的维护频率,从而进一步降低了通信和修复开销。据我们所知,目前还没有基于纠删码存储的系统在数据维护时考虑到数据分类的问题,也没有这方面的研究。最后,我们基于一个开源的分布式文件系统(采用了纠删码冗余方式),实现了原型系统,并提出了一种层次化的实现方法,通过这种方法,大大缩短了原型系统的开发时间,并通过模拟和实际测试的方法对算法和原型系统进行了验证,测试结果表明本文提出的策略是有效的。
[Abstract]:With the rise of cloud storage, big data and other technologies, the survivability of data is paid more and more attention, and the technology of data fault tolerance is the main method to ensure the survivability of data. The current data fault-tolerant technology is mainly implemented by redundant methods, including replica and erasure code. Erasure code is much more efficient than replica in storage efficiency, but it has the problem of large data maintenance cost. In this paper, the problem of data maintenance overhead of erasure code scheme is studied, and two major problems: 1: 1 are put forward. The overhead of maintenance communication between nodes is too high. Data recovery computation is too expensive. In view of these two problems, we put forward the solution. First of all, this paper proposes a passive detection and repair algorithm from the point of view of data reading. The main idea of this method is to use the normal reading bandwidth of the system to detect the data. If the data needs to be repaired, the normally decoded data can be cached locally. When the data detected by this method is repaired, it is not necessary to detect the data again. Download and decode data, reducing communication and repair overhead. As far as we know, there is no such method at present. Secondly, from the point of view of system reliability, an adaptive detection algorithm is proposed to reduce data repair and communication overhead. The main idea of this method is to adjust the frequency of data maintenance according to the different reliability of the system. The main difference between this method and other existing methods is that the data maintenance frequency changes dynamically according to the reliability of the system, which reduces the maintenance frequency and ensures the availability of the data. Thirdly, a tolerance selection algorithm is proposed from the point of view of data classification. The main idea of this method is to realize different maintenance frequency for different data and reduce the maintenance frequency of infrequently accessed data according to the different data access modes (in actual storage system, there are great differences in data access patterns [64]). Thus, the communication and repair costs are further reduced. As far as we know, there is no system based on erasure code storage to consider the problem of data classification in data maintenance, and there is no research in this field. Finally, we implement the prototype system based on an open source distributed file system (using erasure code redundancy method), and propose a hierarchical implementation method. By this method, the development time of the prototype system is greatly shortened. The algorithm and prototype system are verified by simulation and actual test. The test results show that the proposed strategy is effective.
【学位授予单位】：国防科学技术大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333;TP309

【参考文献】