元数据服务器集群负载均衡技术的研究

发布时间：2018-08-11 11:30

【摘要】：随着信息技术的不断发展,全球信息存储量正在以57%的复合年增长率膨胀。大量数据密集型应用对现有的网络存储架构提出了更高的要求。对象存储技术同时具备传统SAN快速访问的特点和NAS跨平台数据共享和强安全性的优势。同时对象存储设备本身具备的智能管理功能。因此,对象存储系统具有高性能、可扩展、高可用性和强安全性等特点,是未来网络存储的发展方向。元数据服务器(MetaData Server, MDS)是对象存储系统的重要组成部分,在对象存储系统中的主要功能包括：管理文件系统名字空间、文件／目录和对象物理存储位置之间的映射、用户访问控制等。因此,MDS集群的处理能力是整个对象存储系统高效运作的重要一环。本文分析了当前主流网络存储架构中元数据服务器集群的负载均衡方案。针对元数据服务器性能异构造成的集群负载不均衡问题,本文提出了一种基于权重的元数据服务器集群负载均衡策略(Weight-based Metadata Server Cluster Load Balancing Strategy, WLBS),并对其在Ceph框架中进行实现。本文的主要研究和开发内容包括以下四个方面： (1)在分析对象存储与传统网络存储技术的基础上,总结了MDS负载均衡技术所涉及的包括元数据分区和服务器热点评价两个关键问题； (2)提出了元数据服务器热点评测模型,主要解决准确获取服务负载状况的问题。该模型在传统计数法的基础上增加了对MDS性能值的考虑,提出了基于操作时延的MDS性能模型。该性能模型避免了传统“白盒”性能模型量化资源影响力的难题,准确评估服务器的当前性能值； (3)提出了一种基于权重负载均衡策略,主要解决MDS性能异构情况下的负载均衡问题。该策略根据(2)中热点评测模型获取节点当前的负载状况,并计算出集群的平均负载水平。为了实现负载与性能相对应的目标,高于平均负载水平的MDS会主动以目录子树迁移方式进行负载迁移,将一部分目录子树迁移到低于平均负载水平的MDS,最终实现异构MDS集群的负载均衡； (4)搭建实验平台,并对策略进行实验评测。对WLBS是否具备对异构MDS集群负载进行调整而使负载均衡化和保持MDS集群处理能力进行验证。大量实验证明本文提出的负载均衡策略可以在异构环境下使元数据服务器集群保持负载均衡,提高了系统的元数据处理能力。
[Abstract]:With the development of information technology, global information storage is expanding at a compound annual growth rate of 57%. A large number of data-intensive applications put forward higher requirements for the existing network storage architecture. Object storage technology has the characteristics of traditional SAN fast access and NAS cross-platform data sharing and strong security advantages. At the same time, the object storage device itself has the intelligent management function. Therefore, object storage system has the characteristics of high performance, scalability, high availability and strong security, which is the development direction of network storage in the future. Metadata server (MetaData Server, MDS) is an important part of object storage system. Its main functions in object storage system include managing file system namespace, mapping file / directory and object physical storage location. User access control, etc. Therefore, the processing capability of MDS cluster is an important part of the efficient operation of the whole object storage system. This paper analyzes the load balancing scheme of metadata server cluster in current mainstream network storage architecture. Aiming at the problem of cluster load imbalance caused by heterogeneous metadata server performance, this paper proposes a load balancing strategy (Weight-based Metadata Server Cluster Load Balancing Strategy, WLBS),) for metadata server cluster based on weight and implements it in the framework of Ceph. The main research and development contents of this paper include the following four aspects: (1) based on the analysis of object storage and traditional network storage technology, This paper summarizes two key issues involved in MDS load balancing technology, including metadata partition and server hot spot evaluation. (2) A metadata server hot spot evaluation model is proposed, which mainly solves the problem of accurately obtaining service load status. Based on the traditional counting method, the MDS performance value is considered in this model, and the MDS performance model based on operation delay is proposed. The performance model avoids the problem of quantifying the resource influence of the traditional "white box" performance model and accurately evaluates the current performance value of the server. (3) A load balancing strategy based on weight is proposed. It mainly solves the load balancing problem in the case of heterogeneous MDS performance. According to the hot spot evaluation model in (2), the current load status of the node is obtained, and the average load level of the cluster is calculated. In order to achieve the goal of corresponding load and performance, MDS with higher than average load level will actively migrate the load in directory subtree mode. Part of the directory subtrees are migrated to MDSs which are below average load level, finally the load balance of heterogeneous MDS cluster is realized. (4) the experimental platform is built, and the policy is evaluated experimentally. This paper verifies whether WLBS has the ability to adjust the load of heterogeneous MDS cluster to balance the load and maintain the processing ability of MDS cluster. A large number of experiments show that the proposed load balancing strategy can keep the load balance of metadata server cluster in heterogeneous environment and improve the metadata processing ability of the system.
【学位授予单位】：杭州电子科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP368.5

【参考文献】