一种文件路径与属性信息分离的分布式元数据组织方法

发布时间：2018-03-16 03:10

本文选题：元数据　切入点：元数据组织　出处：《华中科技大学》2016年硕士论文　论文类型：学位论文

【摘要】：随着大数据时代的到来,面向大数据的存储系统纷纷出现。不断增长的数据量,使得集中式元数据管理系统的负担越来越重,逐渐成为大数据存储的瓶颈。为此,人们提出了多种分布式元数据管理方法,但由于元数据的结构类型复杂多样,目前尚没有一种方法能够同时改善元数据管理的性能和扩展性。提出了一种文件路径和属性信息分离的分布式元数据组织方法。将元数据组织成目录索引和元数据属性信息两个部分,通过构建目录索引,将元数据以目录或小于目录为单位划分到不同的桶(Bucket)内,再根据元数据服务器集群的负载情况将桶指派到不同的元数据服务器上。方法利用目录索引和桶提高元数据的管理性能;通过构建目录索引时考虑集群负载情况,实现元数据管理的可扩展性。此外,提出基于该方法的元数据位置缓存策略,策略解决了位置缓存信息不一致的问题,缩短了元数据管理的流程。测试结果表明,提出的方法能获得较高的管理性能,特别适合高并发的情况;具有良好的可扩展性和较好的访问局部性,而且可以不限制目录的大小;避免了重命名元数据造成的不必要的迁移。与集中式元数据管理方法对比,方法采用单一元数据服务器时,元数据的创建、查询等操作性能都有了数倍的提升。
[Abstract]:With the arrival of big data's era, the storage system for big data appeared one after another. The increasing amount of data makes the burden of centralized metadata management system become more and more heavy, and gradually becomes the bottleneck of big data storage. A variety of distributed metadata management methods have been proposed, but because of the complexity and diversity of the structure of metadata, At present, there is no method to improve the performance and scalability of metadata management simultaneously. A distributed metadata organization method, which separates file path and attribute information, is proposed. The metadata is organized into directory index and metadata. According to two parts of attribute information, By building a directory index, the metadata is divided into different buckets in directories or smaller than directories. Then according to the load of metadata server cluster, the buckets are assigned to different metadata servers. Methods Directory index and bucket are used to improve the management performance of metadata. In addition, a metadata location caching strategy based on this method is proposed, which solves the problem of inconsistent location cache information and shortens the process of metadata management. The test results show that, The proposed method can achieve high management performance, especially suitable for high concurrency, have good scalability and good access locality, and can not limit the size of the directory. Compared with centralized metadata management method, when using single metadata server, the operation performance of metadata creation and query has been improved several times.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【参考文献】