蓝鲸元数据服务器集群的细粒度负载迁移

发布时间：2018-04-13 09:27

本文选题：元数据服务器 + MDS集群　；参考：《计算机研究与发展》2014年S1期

【摘要】：大数据应用对信息系统的底层存储提出了极大挑战,其首选方案为元数据服务器(metadata server,MDS)集群架构.MDS集群系统为了实现负载均衡,采用的基本机制为元数据服务的负载迁移.当前主流方案存在迁移时间长,迁移中写相关元数据访问需要阻塞的问题.提出细粒度的元数据服务负载迁移方案,由迁出端恢复迁移机制和细粒度控制迁移机制2部分组成.在迁移过程中,以文件为粒度在迁入端重建迁出端的元数据相关状态结构,并根据被访问状态的类型选择在迁出端或迁入端进行立即响应,而非阻塞操作直到迁移完成.在蓝鲸MDS集群系统上实现了此方案,测试和分析表明,在10 000规模的迁移场景中,迁移时间控制在800ms以下,元数据或状态写入延迟最高为215ms.与系统原方案中90s的固有延时相比,此方案有效解决了MDS集群中负载迁移时间过长的问题,并且极大地降低了迁移过程对正常元数据访问的影响.
[Abstract]:Big data's application poses a great challenge to the underlying storage of information system. The preferred scheme is metadata server / MDS cluster architecture. In order to achieve load balancing, the basic mechanism adopted by big data is the load migration of metadata service.The current mainstream scheme has the problem of long migration time and the problem of blocking the access to the relevant metadata in the migration.A fine-grained load migration scheme for metadata service is proposed, which consists of two parts: the migration mechanism of migration end recovery and the mechanism of fine-grained control migration.In the migration process, the file granularity is used to reconstruct the metadata correlation state structure of the migration end, and according to the type of the accessed state, it is selected to respond immediately at the outgoing or incoming end, instead of blocking the operation until the migration is completed.The scheme is implemented on the blue whale MDS cluster system. The test and analysis show that the migration time is controlled below 800ms and the metadata or state write delay is up to 215msin the 10 000 scale migration scenario.Compared with the inherent delay of 90 s in the original scheme of the system, this scheme effectively solves the problem of long load migration time in MDS cluster, and greatly reduces the impact of migration process on normal metadata access.
【作者单位】：中国科学院计算技术研究所;中国科学院大学;
【基金】：国家“九七三”重点基础研究发展计划基金项目(2011CB302304) 国家“八六三”高技术研究发展计划基金项目(2011AA01A102,2013AA013205) 中国科学院战略性先导专项课题(XDA06010401) 中国科学院重点部署基金项目(KGZD-EW-103-5(7))
【分类号】：TP368.5

【参考文献】