容错存储系统校验更新及修复优化技术研究

发布时间：2018-01-13 01:13

本文关键词：容错存储系统校验更新及修复优化技术研究　出处：《中国科学技术大学》2017年硕士论文　论文类型：学位论文

【摘要】：近年来随着数据的爆炸式增长,数据的存储规模越来越大,传统的单机系统已经无法满足高速增长的数据存储需求。分布式存储系统使用大量廉价商用服务器通过网络互联,可以提供极强的服务能力和扩展能力。然而,随着集群规模的变大、存储设备的增多,存储节点失效已不是偶然事件。因此,分布式存储系统对数据的可靠性要求尤为突出。分布式存储系统常用的容错技术有两种,多副本技术和纠删码技术。纠删码技术相对于多副本技术在提供相同容错能力的前提下,所需的存储开销更小,因此被存储系统广泛采用。纠删码技术通过对数据切块,计算并存储一定的冗余数据,当发生不超过系统容错能力的节点失效时,可以利用存活节点的数据恢复出失效节点的数据。所以,为保证部署纠删码的存储系统的可靠性,在数据块更新的同时需要更新与之关联的校验数据;同时,当存储校验数据的节点失效时,快速的故障数据恢复对保证分布式存储系统的可靠性也同等重要。本文主要研究了部署纠删码的存储系统校验更新及修复优化问题,主要研究内容与贡献如下:(1)校验数据更新优化研究随着业务规模的增长,已有的存储系统往往无法满足上层应用更大的存储容量和更高的访问带宽需求。当应用需求超过存储系统的负荷时,需要增加新的存储介质到已有的系统中,并将部分数据迁移到新的存储设备上,以均衡负载和提供更大的访问带宽。然而,数据迁移会造成大量校验更新,由此给系统造成额外负载。本文针对扩容过程中的校验数据更新问题,提出了一种优化策略EPU(Efficient Parity Update Algorithm)。首先,EPU根据用户请求访问情况调整扩容顺序,以最大化利用用户请求节省校验更新引发的I/O;其次,EPU通过比较不同校验更新方式的系统开销,选择最佳的校验更新方式,并尽可能的聚合校验更新I/O以减小系统开销。相比于传统扩容方法采用固定的校验更新方式,使用EPU,既可以减少扩容过程引发的系统带宽开销,也能加快扩容过程的完成。(2)校验数据故障恢复优化研究分布式存储系统的规模越来越大,有的甚至达到几百上千台集群规模。庞大的节点数目造成节点故障频发,当发生节点失效时,需要将失效节点修复,以维持数据的可用性。并且,数据的修复速度应尽可能的快,因为如果修复时间过长,修复期间有可能发生更多节点失效,若失效节点数量超过系统的容错能力,将造成数据的永久丢失。因此,数据的快速修复对于系统的可靠性至关重要。本文提出了一种基于局部校验和全局校验相结合的新型编码ESRC(Efficient Single-failure Recovery Code)码,该编码可以高效地修复单个节点故障,与现有的常用纠删码相比,ESRC码可以提高单个校验节点的修复效率,并维持较低的存储开销。
[Abstract]:In recent years, with the explosive growth of data, data storage increasingly large scale, the traditional single machine system has been unable to meet the needs of the rapid growth of data storage. The distributed storage system using a large number of cheap commercial server through the network, can provide strong service capabilities and scalability. However, with the increase of the cluster size becomes large storage device, storage node failure is not accidental. Therefore, distributed storage system reliability requirement of data is particularly prominent. Fault tolerant technology commonly used in distributed storage system has two kinds, the multi technology and erasure codes. Erasure codes with respect to multi copy technology in the premise of providing the same fault tolerance, storage overhead required, therefore has been widely used in storage systems. Erasure codes based on data cubes, calculate and redundant data storage will not occur when. Nodes over the ability of fault failure, recovery can use survival data node failure node data. So, in order to ensure the correct deployment reliability storage system delete code of the data at the same time in block updates need to check update data associated with it; at the same time, when the node storing the check failure data, fault data a quick recovery to ensure the reliability of distributed storage system is equally important. This paper mainly studies the deployment of erasure code storage system check update and repair optimization problems, the main research contents and contributions are as follows: (1) study with the growth in business scale optimization and update the verification data storage system has been unable to meet the storage capacity of the upper application often more and more high access bandwidth demand. When the application needs more than storage system load, the system need to add new storage medium to existing, and Some of the data migration to the new storage device, in order to balance the load and provide greater access bandwidth. However, data migration will cause a large number of parity updates, thus to the system caused by the additional load. Based on the calibration data expansion process update problems, proposes an optimization strategy of EPU (Efficient Parity Update Algorithm) EPU. First, according to the user request access to adjust the expansion in order to maximize the user requests to save check caused by the update of I/O EPU; secondly, through the system overhead check update of check, choose the best way of updating, and as far as possible to reduce the I/O polymerization check update overhead compared to the traditional expansion. Using the method of fixed parity update methods, the use of EPU, can reduce the expansion process caused by system bandwidth overhead, can also speed up the expansion process. (2) check failure data Recovery optimization of distributed storage system increasingly large scale, some even up to hundreds of thousands of Taiwan. The number of nodes in large scale cluster nodes caused by frequent faults, when node failure, node failure will need repair, in order to maintain the availability of data. And, according to the number of repair speed should be as fast as possible, because if the repair time is too long, during the repair may occur more node failure, if the failure tolerance of node number over the system, will cause permanent data loss. Therefore, the reliability of the data for the rapid repair of critical systems. This paper proposes a new encoding ESRC local and global parity check based on the combination of Efficient (Single-failure Recovery Code) codes, the encoding can effectively repair single node failure, compared with the existing common erasure codes, ESRC codes can improve the single check node The repair efficiency and low storage cost are maintained.

【学位授予单位】：中国科学技术大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【相似文献】