高性能计算中并行文件系统可扩展元数据服务的研究

发布时间：2018-01-05 15:16

本文关键词：高性能计算中并行文件系统可扩展元数据服务的研究　出处：《华中科技大学》2013年硕士论文　论文类型：学位论文

【摘要】：随着计算能力的增强、应用课题规模和复杂度的增加，高性能计算机对并行文件系统性能要求越来越高，在海量小文件频繁创删和大规模并发I/O操作的应用场景中，文件系统元数据吞吐率成为限制其性能的关键因素。针对I/O转发架构，提出了一种基于元数据代理的高可扩展元数据服务的方案，结合作业调度系统，将文件系统元数据请求分散到多个元数据代理上，加速作业的文件系统元数据访问。并行I/O是高性能计算系统中主要的I/O使用方式，可分为单数据流模式和共享文件模式。其中单数据流模式给元数据服务带来很大负载，元数据可扩展性方案面向该I/O模式提出的，主要包括元数据代理MDDS(MetaData Delegation Service)和基于MDDS的作业调度两部分。元数据代理基于Lustre元数据集群架构CMD（ClusterMetaData）实现的，通过降低多个元数据服务间的耦合度，保证元数据集群的高可用性；使用目录子树方式管理元数据代理空间，避免跨节点目录引入的分布式原子操作的复杂性和低效性；实现元数据迁移避免跨元数据服务器间文件迁移造成的数据对象移动；通过元数据代理动态增删机制实现元数据代理的灵活部署。针对高性能计算中I/O转发架构作业调度特点，，提出基于元数据代理的两种作业调度策略——单作业独占单元数据代理调度和多作业共享多元数据代理调度。前者实现了对传统作业I/O访问模式的支持，避免了多作业间元数据竞争；后者将单个作业的元数据访问分散到多个元数据代理上，实现了作业内的元数据负载均衡。在116台存储服务器上对元数据代理进了测试，同时通过模拟I/O转发架构中作业的文件系统元数据访问负载对两种作业调度方法进行评估。结果表明，元数据代理提供了拟线性的元数据性能，在大规模的环境中较CMD方案有较好的扩展性；两种调度方式有效分散了作业元数据的负载，改善了高性能计算中的元数据瓶颈问题。
[Abstract]:With the increase of computing power, the application project size and complexity increase, the high performance computer is the higher of the parallel file system performance requirements and application scenarios in massive small files and delete a frequent large-scale concurrent I/O operations, file system metadata throughput become the key factors for its performance. For I/O forwarding architecture. Proposes an extended metadata service based on metadata agent high scheme, combined with the scheduling system, the metadata file system requests are distributed to a plurality of metadata on the proxy, accelerate the file system meta data access operation.
I/O is a high performance parallel computing is used mainly in the I/O system can be divided into a single data stream model and file sharing mode. The single data stream model for metadata service great load metadata scalability scheme for the I/O mode, including MDDS (MetaData Delegation Service metadata agent) and MDDS operation based on the scheduling of two parts. The proxy Lustre metadata metadata cluster architecture based on CMD (ClusterMetaData) to achieve, by reducing the number of metadata service between the coupling degree, ensure high availability of metadata cluster; use the subtree management metadata agent space, to avoid the complexity of cross node directory into distributed atomic operation and low efficiency; metadata avoid cross migration file metadata server migration between data objects caused by the movement; increase delete machine through the metadata dynamic proxy For the realization of flexible deployment metadata agent. For high performance computing I/O forwarding architecture scheduling characteristics, put forward multiple agent scheduling data sharing metadata agent two scheduling strategies: single task scheduling and multi agent exclusive unit data. Based on the former work to achieve the access mode support for the traditional operation I/O, avoid the metadata competition operation; the latter will access a single job metadata distributed to multiple metadata agency, realize the operation in the metadata load balancing.
In the 116 storage server into the test metadata agent, by simulation, I/O file system metadata schema forwarding operation access to evaluate two kinds of load scheduling methods. The results show that the metadata agent provides quasi linear metadata properties, compared with the CMD scheme has good scalability in large scale environments; two scheduling modes effectively disperse the load operation of metadata, improve the metadata bottleneck in high performance computing.

【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【共引文献】