一种基于纠删码的分布式备份技术的研究

发布时间：2018-12-15 13:51

【摘要】：随着互联网技术和信息技术的不断发展，存储系统中的数据量不断增长并呈海量形式发展。传统的依赖本地服务器的存储已经无法满足日益增长的存储需求，因此大规模的基于分布式存储的数据中心正在迅速增长。分布式存储系统中，，为了保障数据的可用性，容灾备份是关键问题之一，这需要使用容错策略来实现系统的高可靠性，特别是存在大量的冷数据的分布式存储系统中，冷数据的备份方式需要特别考虑。针对上述问题，在介绍备份容错策略的基础上，对分布式系统的编码备份技术进行了分析和研究，并提出了一种基于编码的分布式备份方案。通过对现有编码技术的研究，提出了一种基于纠删码的，适用于含大量冷数据分布式系统的一种备份架构。这种架构在保障数据可靠性的同时，能够提高系统的存储利用率，提高系统的负载均衡性，缩短节点失效时的修复时间，增加系统灵活性。在系统设计上，为了使系统的负载更为均衡，将存储节点中的物理节点根据需求分为多个虚拟节点并且排序后均匀分布在圆环结构中，虚拟节点存储用户数据。同时，根据用户需求设计不同的备份服务器，选择存储组，存储组由多个不在同一物理节点的虚拟节点构成，备份服务器中的数据经过分割，纠删码编码后，分散存储到存储组内的虚拟节点上，不同的用户需求可以选择不同的纠删码。处理失效时，数据块通过解码的方式恢复来保证其可靠性，物理节点通过多存储组的并行恢复在保证可靠性的同时，缩短恢复时间。基于设计的模型，对基于纠删码的分布式备份方案进行了仿真测试，通过节点选取和数据修复的测试，验证了编码备份方案能保证数据的可靠性，相较于三副本方案，存储利用率提高36%，较非并行的编码方案，修复时间更短。在测试过程中主要采用了CRS和RS码两种纠删码进行测试，测试表明CRS码的编解码效率比RS码更好，恢复能达到高效率。
[Abstract]:With the development of Internet technology and information technology, the amount of data in storage system is increasing and developing in massive form. Traditional storage relying on local servers can no longer meet the increasing demand for storage, so large-scale data centers based on distributed storage are growing rapidly. In distributed storage system, in order to ensure the availability of data, disaster recovery backup is one of the key problems, which needs to use fault-tolerant strategy to achieve high reliability of the system, especially in distributed storage systems with a large amount of cold data. Cold data backup methods need special consideration. Based on the introduction of backup fault-tolerant strategy, the coding backup technology of distributed system is analyzed and studied, and a distributed backup scheme based on coding is proposed. A backup architecture based on erasure codes is proposed for distributed systems with large amount of cold data. This architecture can not only guarantee the reliability of data, but also improve the storage utilization of the system, enhance the load balance of the system, shorten the repair time when the node fails, and increase the flexibility of the system. In the design of the system, in order to balance the load of the system, the physical nodes in the storage nodes are divided into several virtual nodes according to the requirements and distributed uniformly in the ring structure after sorting. The virtual nodes store user data. At the same time, different backup servers are designed according to the needs of users. The storage group is composed of several virtual nodes that are not in the same physical node. The data in the backup server is divided and erasure code is coded. Different user requirements can select different erasure codes. When processing failure, the data block is restored by decoding to ensure its reliability, and the physical node can shorten the recovery time while guaranteeing the reliability by parallel recovery of multiple storage groups. Based on the designed model, the distributed backup scheme based on erasure code is simulated and tested. Through the test of node selection and data repair, it is verified that the coded backup scheme can guarantee the reliability of data, compared with the three-copy scheme. The storage utilization is increased by 36%, and the repair time is shorter than that of the non-parallel coding scheme. In the process of testing, two kinds of erasure codes, CRS and RS codes, are used to test. The results show that the coding and decoding efficiency of CRS codes is better than that of RS codes, and the recovery efficiency is high.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP309.3;TP333

【参考文献】