分布式键值存储系统高效能数据布局技术研究

发布时间：2018-01-05 06:03

本文关键词：分布式键值存储系统高效能数据布局技术研究　出处：《华中科技大学》2016年博士论文　论文类型：学位论文

【摘要】：分布式键值存储系统是最常用的分布式存储系统之一,目前被广泛部署在大规模数据中心中。然而,大规模数据中心的能耗是一个亟待解决的问题。其中,服务器的能耗占到了整个数据中心的大部分能耗。随着网络数据量的持续增加,数据中心对服务器的需求量会越来越大。因此,服务器能耗已经成为企业数据中心的一个严重问题。目前,很多节能技术采用关闭存储设备或者服务器的方式进行节能。但是,在分布式键值存储系统中关闭服务器是极具挑战性的,因为系统可靠性可能会降低。关闭存储设备带来的可靠性问题已经成为节能研究的关注重点。另外,由于固态盘在性能和能耗方面的优势,以及基于NAND闪存的固态盘存储容量的持续增加和价格的降低,越来越多的固态盘被配置到服务器存储系统中作为缓存或者存储设备。但是,基于闪存的服务器之间的I/O负载是不均衡的,这种不均衡会导致闪存磨损不均衡。磨损不均衡会对整个键值存储集群的可靠性、性能以及寿命产生负面影响。针对基于一致性哈希的分布式键值存储系统的能耗问题,本文首先提出了GreenCHT节能技术,它包括一个分层副本放置策略和一个预测性能耗模式调节器PMS。数据对象的副本被分布到互不重叠的哈希环上而不是随机放置在哈希环的某一些节点上。这种放置策略可以通过关闭部分服务器,使得系统进入不同的能耗模式,而不会影响数据的可用性。另外,能耗模式调节器PMS能够预测I/O负载,并且根据负载的高低波动变化来相应关闭或者开启某些节点层,进入不同的能耗模式.因此,系统的性能和能耗是与系统的负载成比例的。通过关闭不同数目的节点层,系统可以切换到不同能耗模式来满足不同的能耗和性能需求,同时不会影响数据的可用性。为了解决低能耗数据布局中存在的可靠性问题,本文提出了一个分布式的Log-store来维持数据的可靠性：当一部分服务器被关闭后,所有的访问待机服务器的写请求都以一种可靠的数据布局方式被重新分布到其他活动的服务器上。这样的布局方式不仅可以保证数据的可靠性同时还能够维护写请求的并行性.即使是在低能耗状态,Log-store也可以保证副本的冗余度。当系统出现服务器宕机时,系统可以开启一层服务器并且开始数据恢复。这样,即使是在部分节点被关闭的低能耗模式下,系统仍然可以容R-1个节点出错,其中R为副本的冗余度。本文还给出了分布式Log-store处理服务器故障的具体容错方案。另外,本文提出了一个基于闪存耐久性的写转移技术称为EWO,用来均衡不同闪存服务器之间的损耗差异。由于基于数据迁移的损耗均衡算法产生了大量的额外的写请求,这些写请求会导致相当大的额外磨损,EWO的设计是利用闪存存储器异地更新的特性,通过将写请求或者更新请求在不同闪存服务器之间转移或者重定向来减少均衡过程带来的额外损耗。为了将擦除次数均匀的分布到闪存服务器上,EWO首先基于垃圾回收的频率计算出：多少写请求需要从擦除次数比较多的闪存服务器转移到擦除次数比较低的闪存服务器上,然后进行写请求重新分配或者重定向来均衡损耗。为了降低写转移操作带来的元数据开销,EWO采用了一个热切片转移策略HSO,来达到额外的损耗开销和元数据开销之间的平衡。
[Abstract]:Distributed key value storage system is one of the most commonly used in distributed storage systems, has been widely deployed in large-scale data centers. However, consumption of large-scale data centers is an urgent problem to be solved. The server energy consumption accounted for most of the entire data center's energy consumption. With the continued increase in the amount of network data, data the center server demand will become increasingly large. Therefore, the server power consumption has become a serious problem in the enterprise data center. At present, many energy-saving technology using closed storage devices or servers in such way. However, the server is very challenging in the closed distributed key value storage system, because the system reliability may be reduced. Closed storage device reliability problems bring have become the focus of attention on energy conservation. In addition, due to the solid state disk in performance and energy consumption. The advantages, and based on the lower solid disc storage capacity NAND flash and prices continued to increase, more and more solid state disk is configured to cache or storage devices as the server storage system. However, between the flash I/O server load is not balanced based on this imbalance will lead to uneven wear. The wear of flash memory unbalanced reliability of the key storage cluster, have a negative impact on performance and life. To solve the problem of power consumption in distributed key value storage system based on consistent hashing, this paper firstly put forward the energy saving technology of GreenCHT, which includes a copy of a hierarchical replica placement strategy and a predictive power mode regulator of the PMS. data object is distributed to the hash ring do not overlap each other instead of randomly placed at some nodes. This hash ring can be closed part of the service placement strategy Is the system into different modes of energy consumption, but does not affect the availability of data. In addition, the energy consumption mode regulator PMS can predict the I/O load, and according to the fluctuation of load changes corresponding to close or open certain node layer into energy consumption pattern. Because of this, the performance and energy consumption of the system and system the load is proportional. The different number of nodes is closed, the system can be switched to different energy consumption and energy consumption model to meet different performance requirements, but will not affect the availability of data. In order to solve the reliability problems of low energy consumption data in the layout, this paper proposes a distributed Log-store to maintain the reliability of the data: when a part of the server is closed, all write requests access to standby servers are in a reliable data layout was redistributed to other activities of the server . the design can not only ensure the reliability of data can also maintain the parallelism of write requests. Even if it is in the state of low energy consumption, Log-store can also guarantee the redundant copy. When the system server downtime, the system can open a layer of server and start the data recovery. In this way, even if it is a low power consumption mode was shut down in parts of the node, the system can still let R-1 nodes, where R is the redundant copy. This paper also gives the concrete scheme of distributed fault tolerant Log-store server failure. In addition, this paper presents a written transfer flash durability called EWO based on the differences between the different equilibrium, for loss flash server. Because of the loss equalization algorithm based on data migration produced a large number of additional write requests, these write requests will lead to additional considerable wear, EWO. The characteristics of remote meter is updated using flash memory, always reduce the extra loss brought by the equilibrium process of write requests or update request transfer between different servers or Flash retracking. In order to erase times evenly distributed to the flash memory on the server, EWO first based on the garbage collection frequency is calculated: the number of write requests more from the server to erase flash erase times low flash memory on the server, and then write request redirection or re allocation. In order to reduce the loss of balance caused by write transfer operation metadata overhead, EWO uses a hot slice transfer strategy HSO, to achieve between the additional loss cost and metadata cost balance.

【学位授予单位】：华中科技大学
【学位级别】：博士
【学位授予年份】：2016
【分类号】：TP333

【相似文献】