通用可扩展的分布式文件系统性能测试框架研究与实现

发布时间：2018-01-04 22:09

本文关键词：通用可扩展的分布式文件系统性能测试框架研究与实现　出处：《南京大学》2016年硕士论文　论文类型：学位论文

【摘要】：分布式文件系统(DFS, Distributed File System)构成了分布式计算的基础,是如今日益发展的大数据应用的重要核心之一。经过数十年的发展,目前存在种类繁多的分布式文件系统,它们具有不同的设计目标和特点。针对不同的特点,在运行大数据应用时需要不同的集群环境和配置,因此,充分了解分布式文件系统的性能和特性就显得极其重要。一方面,用户需要根据性能测试结果为目标应用选择最合适的分布式文件系统：另一方面,开发者需要一个性能测试框架以进行分布式文件系统的调优工作；进一步地,性能测试能够很好地反映出现有分布式文件系统的瓶颈,为新系统的研究和开发提供指导。现有的分布式文件系统性能测试工具提供的覆盖面有限,并且大都侧重于特定功能的测试,缺乏一个统一的性能测试框架,而实现一个理想的分布式文件系统性能测试框架面临着诸多困难和挑战。首先,该框架必须是通用可扩展的,能够方便地对不同的分布式文件系统进行性能测试；其次,它需要提供灵活的、可定制的测试用例,以满足不同的用户需求和应用特性；最后,测试框架要能够适应不同的分布式环境,并且不能明显地影响分布式文件系统本身的性能,以得到准确可靠的测试结果。本文设计并实现了一个通用可扩展的分布式文件系统性能测试框架,DFS-Perf,以实现对现有的和新的目标分布式文件系统进行性能比较。DFS-Perf支持多种并行测试模式,能够在不同的分布式文件系统上运行一系列典型的测试用例。本文的主要贡献点有：(1)设计并实现了DFS-Perf,一个通用的、高度可扩展的、易使用的分布式文件系统测试框架。DFS-Perf能够以多节点、多进程和多线程的并行方式对不同的分布式文件系统进行性能测试,并且支持添加新的目标系统和测试用例。(2)通过分析现有大数据应用的文件访问模式,设计并实现了一系列典型的测试用例。同时, DFS-Perf支持自动地从运行在分布式文件系统之上的应用中获取文件操作记录,分析其文件访问模式,并生成能够模拟这些应用特性的测试用例。(3)在一个具有40个物理节点,共960个核的集群环境上使用DFS-Perf对四个具有代表性的分布式文件系统,Alluxio、CephFS、GlusterFS以及HDFS进行了性能和可扩展性测试实验。进一步地,通过对比实验验证了DFS-Perf最多只引入了5.7%的额外开销。(4)根据性能测试结果,对比了不同分布式文件系统的设计决策,分析了产生性能瓶颈的具体原因,总结出不同特性对性能造成的影响。同时,也利用DFS-Perf发现和解决了常用分布式文件系统中的性能问题。
[Abstract]:Distributed file system (DFS, Distributed File system) forms the foundation of distributed computing. After decades of development, there are a variety of distributed file systems, which have different design objectives and characteristics. Different cluster environments and configurations are required to run the big data application, so it is extremely important to fully understand the performance and features of distributed file systems. Users need to select the most suitable distributed file system according to the performance test results. On the other hand, developers need a performance testing framework to optimize the distributed file system. Furthermore, performance testing can well reflect the bottleneck of distributed file system and provide guidance for the research and development of new system. The existing distributed file system performance testing tools provide limited coverage. And most of them focus on the test of specific functions, and lack of a unified performance testing framework, and the implementation of an ideal distributed file system performance testing framework is facing many difficulties and challenges. First of all. The framework must be general-purpose and extensible, and it can easily test the performance of different distributed file systems. Second, it needs to provide flexible, customizable test cases to meet different user needs and application features; Finally, the test framework should be able to adapt to different distributed environments and not significantly affect the performance of the distributed file system itself. In order to obtain accurate and reliable test results, this paper designs and implements a general extensible distributed file system performance testing framework named DFS-Perf. To achieve performance comparison between existing and new target distributed file systems. DFS-Perf supports multiple parallel test modes. A series of typical test cases can be run on different distributed file systems. The main contribution of this paper is the design and implementation of DFS-Perf. a general-purpose and highly extensible. The easy-to-use distributed file system testing framework .DFS-Perf can test the performance of different distributed file systems in a multi-node, multi-process and multi-thread parallel manner. And support to add new target system and test case. 2) A series of typical test cases are designed and implemented by analyzing the file access pattern of existing big data application. At the same time. DFS-Perf supports automatically obtaining file operation records from applications running on distributed file systems and analyzing their file access patterns. And generate test cases that can simulate these application features in one of 40 physical nodes. In a 960 core cluster environment, DFS-Perf is used for four representative distributed file systems, such as Alluxio / CephFS. GlusterFS and HDFS are tested for performance and extensibility. Through the contrast experiment, it is verified that DFS-Perf only introduces 5.7% extra cost at most.) according to the performance test results, the design decisions of different distributed file systems are compared. The causes of the performance bottleneck are analyzed, and the effects of different characteristics on performance are summarized. At the same time, the performance problems in distributed file systems are also found and solved by using DFS-Perf.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.52

【相似文献】