高性能小文件应用文件系统Sfs设计与实现

发布时间：2018-07-12 19:52

本文选题：Linux + 文件系统　；参考：《上海交通大学》2013年硕士论文

【摘要】：长期以来，计算机其他部件如CPU，内存等性能快速提升，而磁盘由于其固有的机械运动的限制，性能已难以提高，成为制约系统性能的瓶颈。而在一些特定的应用场景如web应用环境中，保存在web服务器上的文件数量特别多（千万至亿），单个文件比较小（KB级别），现有的文件系统在这类应用中也存在着明显不足。如何更高效地组织管理web服务器上的小文件，提升小文件的I/O性能一直是学术界和产业界研究的热点。本文针对现有文件系统对小文件存储优化作了较为深入的分析和研究，设计并实现了适应小文件应用环境的文件系统Sfs。论文的主要工作如下： 1.对当前IT发展趋势作了概要性的分析，提出了在当前互联网和云时代存储面临的迫切需求和严峻挑战； 2.分析研究现有文件系统的主要结构和部分实现细节，尤为关注其针对小文件的性能优化策略。论文中不仅分析了Linux内核支持的如Ext2、Ext3、Xfs、Reiserfs等本地文件系统，同时更涉及了当今世界著名互联网企业和开源项目组针对其业务特点而设计的存储方案，如Facebook的Haystack，淘宝的Tfs等； 3.详细描述了针对小文件应用环境设计的服务器文件系统Sfs。在磁盘布局方面，Sfs摒弃了传统文件系统按固定大小划分磁盘块的做法，采用磁盘划分卷，卷中动态分块的策略，以适应应用环境中各种大小文件的存储需求；尽量简化文件元数据，简化后的元数据减少了文件读写时的I/O交互次数，也可以在内存中缓存尽量多的文件元数据；在文件系统接口设计上，Sfs提供简单但高效的API，利于文件系统的处理效率，同时为Sfs设计了小对象内存池，进一步优化文件系统处理效率； 4.自行设计文件系统测试工具，分别测试文件系统I/O吞吐率和IOPS，以验证在小文件应用环境中Sfs的有效性。测试结果表明，通过更合理的设计和简化的实现，，Sfs在处理小文件时有着更好的I/O吞吐性能和更高的IOPS。
[Abstract]:For a long time, the performance of other computer components, such as CPU, memory and so on, has been rapidly improved. However, due to the limitation of its inherent mechanical movement, the performance of disk has been difficult to improve, which has become the bottleneck restricting the performance of the system. However, in some specific application scenarios such as web application environment, the number of files stored on web server is especially large (tens of millions to billions), the single file is relatively small (KB level), and the existing file system also has obvious shortcomings in this kind of application. How to organize and manage small files on web server more efficiently and improve the I / O performance of small files has always been a hot topic in academia and industry. This paper analyzes and studies the optimization of small file storage for existing file systems, and designs and implements a file system Sfswhich is suitable for the application environment of small files. The main work of this paper is as follows: 1. This paper gives a brief analysis of the current IT development trend and puts forward the urgent needs and severe challenges faced by storage in the current Internet and cloud age; 2. This paper analyzes the main structure and some implementation details of the existing file system, and pays special attention to its performance optimization strategy for small files. This paper not only analyzes the Linux kernel supporting local file systems such as Ext2OU Ext3XfsU Reiserfs, but also involves the storage schemes designed by the world famous Internet enterprises and open source project teams according to their business characteristics, such as Facebook Haystack, Taobao TFS and so on. 3. The server file system Sfs. designed for small file application environment is described in detail. In the aspect of disk layout, SFS abandons the traditional method of dividing the disk block according to the fixed size of the file system, and adopts the strategy of partition the volume and the dynamic block in the volume, so as to meet the storage requirements of the files of various sizes in the application environment. The simplified metadata can reduce the number of I / O interactions and cache as much file metadata as possible in memory. In the file system interface design, SFS provides simple but efficient API, which is conducive to file system processing efficiency. At the same time, it designs a small object memory pool for SFS to further optimize the file system processing efficiency; 4. A file system testing tool was designed to test the I / O throughput and IOPSs of file systems respectively to verify the effectiveness of SFS in small file application environments. The test results show that SFS has better I / O throughput performance and higher IOPS when processing small files through more reasonable design and simplified implementation.
【学位授予单位】：上海交通大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【共引文献】