云存储环境下分布式文件系统的副本策略研究

发布时间：2018-02-12 17:48

本文关键词： 云存储副本策略负载均衡一致性　出处：《电子科技大学》2013年硕士论文　论文类型：学位论文

【摘要】：随着互联网的高速发展，网络上产生的数据量呈爆炸式的增长，对这些数据的存储成为了计算机领域研究的热点。传统的存储方式，比如存储区域网络和网络附加存储因为其存储容量和性能存在瓶颈，价格昂贵，，不易扩展等原因，应用范围受到限制。云存储采用分布式文件系统为核心，硬件设备价格低廉，可扩展性好。副本技术提高了系统的可靠性、可用性和性能的同时，也带来了负载均衡，网络带宽开销，一致性等问题。本文分析了当前典型的分布式文件系统，结合云存储环境下的业务特征，研究了相应的副本策略，重点考虑系统访问的效率和性能，负载均衡，数据的一致性等问题，并根据这些需求设计和实现了分布式文件系统中的副本管理模块。本文的主要工作包括以下内容：（1）提出了基于一致性哈希的副本放置策略。传统的分布式文件系统副本放置位置主要通过元数据中心服务器存储，当系统并发访问量很大时，元数据服务器将成为系统的瓶颈。基于一致性哈希的副本放置策略很好的解决了文件检索，存储设备扩展和失效的问题。引入虚拟节点映射，极大地减少了系统设备的改变带来的数据迁移。同时对设备按照存储能力增加权重属性，更好的提高了系统的负载均衡。（2）提出了基于文件热度的副本调整策略。该策略以文件的请求次数为基础，结合服务器的负载，动态地调整副本数量，提升了系统的性能和效率。同时，辅以副本压缩策略，对长时间没有访问的文件副本压缩，在保证数据可靠性和可用性的同时，节省了系统的存储空间。（3）提出了基于用户请求的副本一致性策略。该策略在对副本一致性维护的同时，充分考虑到了避免增加系统开销。同时，为了防止长期没有用户访问的文件不一致而使得数据的可靠性降低，副本失去冗余备份的作用，采用定时更新的机制，在系统空闲的时候将系统中副本各版本更新到一致状态。
[Abstract]:With the rapid development of the Internet, the amount of data generated on the network is increasing explosively. The storage of these data has become a hot topic in the field of computer. For example, the storage area network and network additional storage are limited because of the bottleneck of storage capacity and performance, expensive and difficult to expand, etc. Cloud storage uses distributed file system as the core, and the hardware is cheap. The replica technology not only improves the reliability, availability and performance of the system, but also brings some problems such as load balancing, network bandwidth overhead, consistency and so on. In this paper, the typical distributed file system is analyzed, and the corresponding replica strategy is studied based on the business characteristics of cloud storage environment. The system access efficiency and performance, load balance, data consistency and so on are considered. The replica management module in distributed file system is designed and implemented according to these requirements. The main work of this paper includes the following contents:. This paper proposes a replica placement strategy based on consistency hashing. The traditional distributed file system replica location is stored mainly through metadata center server. Metadata server will become the bottleneck of the system. The replica placement strategy based on consistency hash solves the problems of file retrieval, storage device extension and invalidation. It greatly reduces the data migration caused by the change of the system equipment, and increases the weight attribute of the device according to the storage capacity, thus improving the load balance of the system. (2) A copy adjustment strategy based on file heat is proposed. This strategy is based on the number of requests of files, and dynamically adjusts the number of copies in combination with the load of the server, which improves the performance and efficiency of the system. At the same time, it is supplemented by a copy compression strategy. The compression of long time unvisited copies of files saves the storage space of the system while ensuring the reliability and availability of the data. (3) A replica consistency policy based on user request is proposed. This policy takes full account of avoiding system overhead while maintaining replica consistency. At the same time, In order to prevent the file inconsistency which has not been accessed by the user for a long time, the reliability of the data is reduced and the replica loses the role of redundant backup. In order to update each version of the copy in the system when the system is idle, the mechanism of timing update is adopted to update each version of the copy to the same state.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333;TP316.4

【参考文献】