一种跨HDFS集群的文件资源调度机制

发布时间：2018-05-24 19:04

  本文选题：集群文件系统 + 分布式文件系统　； 参考：《计算机学报》2017年09期

【摘要】：集群文件系统作为一种典型的分布式文件系统类型,通过集群内多个节点的协同,消除了单点故障以及性能瓶颈问题,实现了高可用、高性能以及动态负载均衡,并且具有较高的可扩展性,因此常作为实现和提供云存储服务的关键技术之一.该文针对HDFS集群主要局限于同一数据中心内部部署且可扩展性受限的问题,提出一种跨数据中心集群部署的文件资源调度机制和金蝶分布式文件服务KDFS,通过分布式架构再设计,支持多个HDFS集群动态组网协同工作;通过引入文件资源池,屏蔽了不同集群之间的文件差异性,能够面向多应用提供透明服务;通过引入弹性存储与最优存储策略,确保集群资源安全冗余与就近服务的同时提升了集群的存储效率.实验和实践证明,跨HDFS集群的文件资源调度机制不但解决了HDFS集群可扩展性受限的问题,同时通过跨数据中心部署,实现了集群文件异地冗余灾备、跨数据中心负载均衡以及文件就近存取服务,有效地提高了应用使用KDFS存储服务的体验.
[Abstract]:As a typical type of distributed file system, cluster file system eliminates single point fault and performance bottleneck problem, and realizes high availability, high performance and dynamic load balancing through the cooperation of multiple nodes in the cluster. Because of its high scalability, it is often used as one of the key technologies to implement and provide cloud storage services. Aiming at the problem that HDFS cluster is mainly confined to the same data center and its expansibility is limited, this paper proposes a file resource scheduling mechanism and Kingdee distributed file service, which is deployed across the data center cluster, which is redesigned by distributed architecture. By introducing file resource pool, file diversity between different clusters can be shielded, and transparent service can be provided for multiple applications. By introducing flexible storage and optimal storage strategy, we can support dynamic networking of multiple HDFS clusters. The cluster storage efficiency is improved by ensuring cluster resource security redundancy and nearby service. The experiment and practice prove that the file resource scheduling mechanism across HDFS cluster not only solves the problem of limited scalability of HDFS cluster, but also realizes the redundant disaster preparedness of cluster files in different places through cross-data center deployment. Cross-data center load balancing and file access services effectively improve the application experience of using KDFS storage services.
【作者单位】： 金蝶软件中国有限公司大数据云平台部;金蝶软件中国有限公司国家企业互联网服务支撑软件工程技术研究中心;东南大学计算机科学与工程学院;武汉大学软件工程国家重点实验室;
【基金】：国家“九七三”重点基础研究发展规划项目基金(2016YFB1000800) 国家云计算示范工程(发改办高技[2011]2448号) 国家发改委大数据示范工程(发改办高技[2014]648号) 广东省信息产业发展专项(粤经信电软[2015]141号) 广东省领军人才计划([2012]342号);广东省领军人才计划(粤人才[2012]342号)资助 深圳市技术开发项目(FWY-CX20140310010238)~~
【分类号】：TP333
，

本文编号：1930203