大规模分布式文件系统元数据管理子系统的设计与实现

发布时间：2019-03-16 14:37

【摘要】：分布式存储系统是云计算领域的核心技术。随着存储即服务这一云计算商业模式的出现，云端存储的数据量正在迅速增大，人们对数据安全、数据共享、数据管理等方面的要求也开始逐步提高。云存储服务提供商的后台正面临着巨大的存储压力。元数据是分布式存储系统中的一个概念，是描述所存储数据命名空间和属性的信息。由于元数据的性质及其操作方式较普通数据复杂，对性能、并发性、安全性的要求极高，因此，对系统中元数据的管理一直是分布式存储系统设计的重点和难点。本文以自行开发的大规模分布式文件系统C-STORE为背景，详细阐述了其元数据管理子系统的设计与实现。该子系统利用三层散列机制进行架构，将元数据分桶存储并进行负载均衡调度，使系统具备了较好的可扩展性。在存储上，元数据管理子系统充分利用了本地文件系统的高效特性，尽可能的提高各方面性能；在组织上，元数据基于用户进行隔离，为用户的数据安全提供了保障。有效且完备的日志管理和同步被设计出来，满足用户多点登录时文件系统的一致性需求。该子系统还引入了Master-Standby双副本同步机制，保证了系统元数据的可用性。元数据管理子系统通过C++在Linux环境下采用异步事件驱动这一高效网络编程模型实现，，充分利用服务器的软硬件资源，提升了系统的吞吐量。经测试，元数据管理子系统可以充分满足用户对面向公有云存储服务在元数据管理各方面的需求。在性能上，相较于当前业界已经十分成熟且广泛使用的PVFS，优势明显。
[Abstract]:Distributed storage system is the core technology in cloud computing. With the emergence of storage-as-a-service cloud computing business model, the amount of data stored in the cloud is increasing rapidly, and people's requirements for data security, data sharing, data management and so on are gradually increasing. Cloud storage service providers are facing enormous storage pressure behind the scenes. Metadata is a concept in distributed storage system, which describes the namespace and attributes of stored data. Because the nature and operation mode of metadata are more complex than ordinary data, the requirement of performance, concurrency and security is very high. Therefore, the management of metadata in the system has always been the focus and difficulty in the design of distributed storage system. Based on the large-scale distributed file system (C-STORE), this paper describes the design and implementation of metadata management subsystem in detail. The sub-system uses a three-layer hashing mechanism to implement the architecture of metadata storage and load balancing scheduling, which makes the system have better scalability. In storage, the metadata management subsystem makes full use of the efficient characteristics of the local file system to improve the performance of all aspects as far as possible; in the organization, metadata isolation is based on the user, which provides the security of the data of the user. Efficient and complete log management and synchronization are designed to meet the consistency requirements of the file system when users log in at multiple points. The sub-system also introduces Master-Standby dual copy synchronization mechanism to ensure the availability of the metadata of the system. The metadata management subsystem is realized by using asynchronous event-driven network programming model in Linux environment, which makes full use of the software and hardware resources of the server and improves the throughput of the system. After testing, the metadata management subsystem can fully meet the needs of users in metadata management for public cloud-oriented storage services. In terms of performance, compared with the current industry has been very mature and widely used of PVFS, advantages.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】