分布式文件存储系统副本管理方法研究

发布时间：2018-03-14 11:20

本文选题：分布式文件系统　切入点：副本管理　出处：《浙江大学》2013年硕士论文　论文类型：学位论文

【摘要】：随着互联网、物联网等新型应用的蓬勃发展,全球数据呈现爆炸式增长,大数据成为应用系统的特点。分布式文件存储系统通过分布式的架构支持海量文件数据的存储,是非结构化大数据存储的核心技术。而副本管理可增强系统的可靠性和容错性、方便进行系统负载均衡和服务性能调优,是分布式文件存储系统的关键技术之一。论文分析分布式文件存储系统中副本管理的功能和流程,综合考虑了用户访问行为、负载均衡、系统代价等方面,提出一种新型的副本管理方法。论文提出一种基于文件热度的动态副本管理方法。首先提出了一种多频率的文件热度算法,对文件多个周期的访问频率进行综合考虑来定量描述文件热度。同时,对于不同热度的文件,采用不同频率的热度更新策略。基于文件热度值,提出了包括副本数的调整方法,多频率的副本执行机制,副本放置位置的策略以及副本的一致性保证的副本管理方法。实验表明热度算法有效的解决了文件热度的统计周期与统计热度准确性之间的矛盾和用户的突发性访问问题,使得副本管理方法更加合理。论文提出考虑负载均衡的实时副本策略。首先通过动态调整负载指标权重,设计了一种可动态自适应应用程序的负载均衡算法,然后提出了与负载均衡算法相结合的实时副本策略,增加了高热度副本的缓存机制和实时副本调度机制。最后通过实验测试验证了应用程序自适应的动态负载均衡算法中的合理性以及实时副本策略的有效性；两者配合使用对于系统性能有较大提升。论文在钱塘分布式文件存储系统中实现了本文提出的副本管理方法,使系统在吞吐率、响应时间、负载均衡等方面都有了一定的提升。基于此存储系统的应用在JTang云盘,已被100多家企事业单位投入使用,效果良好。
[Abstract]:With the rapid development of new applications such as the Internet of things and the Internet of things, the global data is explosively growing. Big data has become the characteristic of the application system. Distributed file storage system supports the storage of massive file data through the distributed architecture. Replica management can enhance the reliability and fault tolerance of the system, facilitate system load balancing and service performance tuning. It is one of the key technologies of distributed file storage system. This paper analyzes the function and flow of replica management in distributed file storage system, and synthetically considers user access behavior, load balance, system cost, etc. A new replica management method is proposed. In this paper, a dynamic copy management method based on file heat is proposed. Firstly, a multi-frequency algorithm is proposed to quantitatively describe the heat of the file by synthetically considering the access frequency of multiple periods. For the files with different heat degrees, different frequency heat updating strategies are adopted. Based on the file calorific values, an adjustment method including the number of replicas is proposed, and a replica execution mechanism with multiple frequencies is proposed. The experiment shows that the heat algorithm can effectively solve the contradiction between the statistical cycle of document heat and the accuracy of statistical heat and the problem of user's sudden access. Make the copy management method more reasonable. In this paper, a real-time replica strategy considering load balancing is proposed. Firstly, a dynamic adaptive load balancing algorithm is designed by adjusting the load index weight dynamically. Then a real-time replica strategy combined with load balancing algorithm is proposed. The cache mechanism and real-time copy scheduling mechanism of high thermal copy are added. Finally, the rationality of the application adaptive dynamic load balancing algorithm and the effectiveness of the real-time replica strategy are verified by experimental tests. The combination of the two can improve the performance of the system. This paper implements the replica management method in the Qiantang distributed file storage system, which improves the throughput, response time and load balance of the system. The application of this storage system in JTang cloud disk. Has been more than 100 enterprises and institutions put into use, the effect is good.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】