BDCode:一种面向大数据存储系统的纠删码算法

发布时间：2018-12-19 20:09

【摘要】：针对含有大量数据的大数据存储系统,提出了一种基于编码技术的面向大数据备份的优化算法(BDCode).该算法通过对不同编解码服务器设置不同的虚拟节点存储组来保证系统的可用性,节点和数据块的并行解码计算提高了系统中数据损坏时的恢复效率.实验表明,所提出基于编码的大数据系统备份机制可以提高系统的存储利用率,并行解码方式的引入能加速减少数据损坏时的恢复时间,并能达到零号的系统负载均衡;此外不同的用户设置不同的编码参数,增加了大数据系统的鲁棒性.实验通过设置不同的数据块m和校验块k的比例来提升利用率,并行解码速度相比以前的串行提高近两倍.使用BDCode比CRS编码效率平均高36.1%,解码效率平均高19.3%;比RS码编码效率平均高58.2%,解码效率平均高33.1%.
[Abstract]:Aiming at big data storage system with a large amount of data, this paper presents an optimized algorithm (BDCode). Based on coding technology for big data backup. The algorithm guarantees the availability of the system by setting different virtual node storage groups to different codec servers. The parallel decoding computation of nodes and data blocks improves the recovery efficiency when the data in the system is damaged. Experimental results show that the proposed system backup mechanism based on coding can improve the storage utilization of the system, and the introduction of parallel decoding can accelerate the reduction of recovery time when the data is damaged, and achieve zero load balance. In addition, different users set different coding parameters, increasing the robustness of big data system. The efficiency of parallel decoding is improved by setting the ratio of different data block m and check block k, and the parallel decoding speed is nearly twice as high as the previous serial decoding speed. The average efficiency of using BDCode is 36.1% higher than that of CRS, the decoding efficiency is 19.3um, the coding efficiency is 58.2% higher than that of RS, and the decoding efficiency is 33.1% higher than that of RS.
【作者单位】：九江学院信息科学与技术学院;广东工业大学计算机学院;平安科技(深圳)有限公司;
【基金】：国家自然科学基金(61462048);国家自然科学基金广东省联合基金重点项目(U2012A002D01) 江西省科技厅科技项目(GJJ151081)资助
【分类号】：TP333

【相似文献】