分布式文件系统的负载均衡策略研究

发布时间：2018-03-16 11:53

本文选题：分布式文件系统　切入点：DHT　出处：《电子科技大学》2014年硕士论文　论文类型：学位论文

【摘要】：分布式文件系统从最初的网络文件系统发展至今,逐渐形成现阶段的云存储。它是在云计算(Cloud Computing)概念上延伸和发展出来的一个新概念。在大集群、动态的分布式文件系统中,随着文件数量、文件访问次数的增加,中心节点将成为整个系统的性能瓶颈。因此,减少中心节点在分布式文件系统中的依赖性是很重要的问题。此外,还有一种基于分布式哈希表(DHT)的分布式文件系统,其中不存在中心节点,并且文件根据DHT算法,均匀分布于系统中。但是,伴随着文件修改、热点出现,系统将无法保持负载均衡状态。负载均衡在分布式文件系统中是很重要的问题。系统负载状态影响着集群存储利用率和网络吞吐量。同时,已均衡的集群能够有效避免热点的出现,提高系统响应速度。因此,本论文将对基于DHT的分布式文件系统负载再均衡问题进行深入的研究。通过对现有负载再均衡算法研究与分析,其执行过程中,由于负载信息获取的局部性,导致在数据迁移过程中出现抖动的现象,并且数据迁移过程中会导致额外高负载节点的出现,从而影响负载均衡效率。因此,本论文通过扩大随机样本节点的个数,使计算出的估计值更加贴近实际值;在选择后继节点的过程中,采取双向延伸的方式,找出满足条件的后继节点,以此减少多余的数据迁移操作。该算法有效地解决了抖动现象的发生,避免了额外高负载节点的出现。同时,副本管理和节点选择也是分布式文件系统中的重要问题。文件系统中的副本分发策略与DHT算法中的文件分发策略不兼容。本论文通过采用软链接方式将副本存放位置重定向,保证两种策略的兼容。结合负载再均衡算法提出节点选择策略,避免数据迁移过程中,多个低负载节点同时选择同一个高负载节点的冲突问题。在负载均衡算法的应用中,使用Space-filling Curve将节点物理位置与逻辑位置对应起来,同时利用节点容量使用率将异构节点集群转换为同构集群。综上所述,本论文提出一种ILR(Improved Load Rebalancing)算法。采用Matlab数学软件和Chord仿真程序对分布式文件系统的负载均衡算法进行实验仿真。仿真实验生成数据后,将本论文提出的ILR算法与现有的负载在均衡算法从累积分布函数、节点负载状态、数据迁移次数和负载信息交换次数四个方面进行比较与分析。结果验证了ILR算法的有效性。
[Abstract]:Distributed file system is a new concept extending and developing from the original network file system to present stage cloud storage. In large cluster, dynamic distributed file system, it is a new concept that extends and develops in cloud computing. As the number of files and the number of file visits increase, the central node becomes a performance bottleneck for the system. Therefore, reducing the dependency of the central node in a distributed file system is an important issue. There is also a distributed file system based on distributed hash table (DHT), in which there is no central node, and files are evenly distributed in the system according to DHT algorithm. The system will not be able to maintain load balancing. Load balancing is an important issue in distributed file systems. System load status affects cluster storage utilization and network throughput. A balanced cluster can effectively avoid hot spots and improve system response speed. In this paper, the load rebalancing problem of distributed file system based on DHT will be deeply studied. Through the research and analysis of the existing load rebalancing algorithms, during the execution process, due to the localization of load information acquisition, The phenomenon of jitter occurs in the process of data migration, and the appearance of extra high-load nodes in the process of data migration, which affects the efficiency of load balancing. Therefore, this paper expands the number of random sample nodes. The estimated value is closer to the actual value, and in the process of selecting the successor node, a two-way extension is adopted to find out the successor node that satisfies the condition. In order to reduce the redundant data migration operation, the algorithm can effectively solve the jitter phenomenon and avoid the appearance of extra high load nodes. At the same time, Copy management and node selection are also important problems in distributed file system. The copy distribution strategy in file system is not compatible with the file distribution strategy in DHT algorithm. This paper proposes a node selection strategy combined with the load rebalancing algorithm to avoid the problem of multiple low-load nodes choosing the same high-load node at the same time in the process of data migration. Space-filling Curve is used to match the physical location and logical location of nodes, and the heterogeneous node cluster is transformed into isomorphic cluster using node capacity utilization. In this paper, a ILR(Improved Load rebalancing algorithm is proposed. The load balancing algorithm of distributed file system is simulated by Matlab software and Chord simulation program. The ILR algorithm proposed in this paper is compared with the existing load balancing algorithm from four aspects: cumulative distribution function, node load state, data migration times and load information exchange times. The results verify the effectiveness of the ILR algorithm.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP333

【共引文献】