海量存储系统中并行文件系统的测试与优化

发布时间：2018-01-15 04:19

本文关键词：海量存储系统中并行文件系统的测试与优化　出处：《上海交通大学》2012年硕士论文　论文类型：学位论文

【摘要】：海量存储系统(Mass Storage System)是为存储海量数据而研制的存储系统,其本质特征在于该存储系统的可扩展性,即能在扩展系统容量和性能的同时,不增加系统管理的复杂性。并行文件系统(Parallel File System)是由一组节点(Node)组成的,这组节点通过相互之间的通信与协作,以更快的速度完成一项大规模的计算任务。并行文件系统是构筑高性能海量存储系统的基石,所以无论是在高性能计算还是在云存储都有着重要的研究意义,因此近年来得到广泛的关注和研究。并行文件系统的测试,大致包括一致性测试、功能测试、压力测试、性能测试、推测性测试等,其中性能测试其中的重点。并行文件系统性能的测试测试的指标大致有两个,一个是吞吐量,衡量大文件I/O读写能力;另一个是IOPS,衡量小文件读写能力。常用优化并行文件系统性能的策略有数据预取,数据缓存、合并小顺序I/O、条带化等技术。缓存指利用缓冲区保存最近访问过的文件内容,以提高对文件的访问效率。预取是指在实际使用内存时提前将数据读取到内存中,从而提高访问效率。合并小顺序I/O,则是将若干个小文件的操作聚合成一个大文件的操作,从而减小磁盘操作的次数。条带化是把连续的数据块分割成相同大小的数据块,将每段数据分别写入到阵列中不同的磁盘。本文主要研究并行文件系统的测试方法,以及优化性能的策略。研究内容主要包括:研究并行文件系统性能测试的标准及方法、学习改进文件系统性能的测试的方法、以GlusterFS为例验证各优化策略的效果。实验证明GlusterFS中的缓存策略改善了超过10MB/s的读性能,预取策略的两个方法合计提供了约15MB/s的读性能。
[Abstract]:Mass Storage system is a storage system developed for storing mass data. Its essential feature lies in the scalability of the storage system. That is, it can expand the capacity and performance of the system at the same time. Parallel file system parallel File system is composed of a set of nodes. This group of nodes completes a large-scale computing task with faster speed through mutual communication and cooperation. Parallel file system is the cornerstone of constructing high performance mass storage system. Therefore, both in high performance computing and cloud storage have important research significance, so in recent years, it has received extensive attention and research. The test of parallel file system includes conformance test, function test, stress test, performance test, conjectural test and so on. Among them, the key points of performance test. There are two indexes of parallel file system performance test, one is throughput, which measures the reading and writing ability of large file I / O; The other is IOPS, which measures the ability to read and write small files. The commonly used strategies to optimize the performance of parallel file systems include data prefetching, data caching, merging small order I / O, striping and so on. Caching is the use of buffers to save the contents of recently accessed files in order to improve the efficiency of accessing files. Prefetching refers to reading data into memory ahead of time when memory is actually used. In order to improve access efficiency, merging small order I / O is the operation of aggregating several small files into one large file. In order to reduce the number of disk operations, striping is to divide the continuous data blocks into blocks of the same size and write each piece of data to a different disk in the array. This paper mainly studies the testing method of parallel file system and the strategy of optimizing performance. The research contents include: research the standard and method of parallel file system performance test. Learn how to improve file system performance testing. GlusterFS is taken as an example to verify the effectiveness of the optimized strategies. Experiments show that the cache policy in GlusterFS improves the reading performance of more than 10MB / s. The two methods of the prefetching strategy together provide about 15 MB / s read performance.
【学位授予单位】：上海交通大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【参考文献】