基于HDFS的海量小文件存储系统设计与实现

发布时间：2018-06-10 00:19

本文选题：海量小文件存储 + 分布式文件系统　；参考：《国防科学技术大学》2012年硕士论文

【摘要】：近年来，企业和个人数据都呈现爆炸性增长的趋势。谷歌首席执行官EricSchmidt表示，现在全球每两天所创造的数据量等同于从人类文明至2003年间产生的数据量的总和。如何存储海量的数据，成为当前存储系统所面临的巨大挑战。传统集中存储方式已经满足不了数据存储的需求，，于是出现了用于大规模数据存储的分布式文件系统，如Google File System（GFS）、Hadoop File System（HDFS）、PVFS、Luster等。这些分布式文件系统具有良好的可扩展性和容错特性，能够满足海量数据存储的需求。但是在很多应用场合除了要求支持海量大文件的存储，还需要支持海量小文件的存储。虽然GFS、HDFS等分布式文件系统能够满足大文件的高效存储，但在存储海量小文件时，效率却很低。针对此问题，工业界和学术界提出了很多方法，但普遍存在性能低，系统可靠性不高，不能高效存储小文件元数据等问题。针对这些挑战，本文设计实现了一种基于HDFS的海量小文件存储系统。该系统的主要设计思想是，在HDFS现有的目录树结构下，将一个文件夹内的小文件，打包成一个大文件进行存储，该文件称为小文件数据文件。同时生成小文件索引，记录小文件在对应数据文件中的位置。本文设计和实现的基于HDFS的海量小文件存储系统是可扩展、高容错、分布式的海量小文件存储集群系统。本文提出小文件聚合存储技术通过将小文件数据存储在HDFS数据文件中，实现数据的分布式存储和容错；同时提出小文件分布索引管理技术将索引分布到各个数据节点管理，解决了单一元数据节点在存储海量小文件成为瓶颈的缺点；设计的海量小文件存储系统索引容错机制通过对索引进行容错，降低小文件丢失的风险；通过在单个目录下创建多个多数据文件，解决访问同一目录下小文件冲突的问题。在以上基础上，系统在客户端缓存用户常用到的小文件索引位置及数据文件流的信息，提高系统的文件访问的效率。通过实验表明，该系统小文件读写延迟、吞吐率与不增加小文件支持的原生HDFS相比有了很大的提高。并且，该系统能够有效解决海量小文件存储元数据过于庞大的问题，且通过索引容错机制，提高了该系统的可靠性。
[Abstract]:In recent years, both corporate and personal data have shown an explosive growth trend. Google CEO Eric Schmidt said the amount of data created every two days in the world is now equivalent to the amount of data generated between human civilization and 2003. How to store huge amounts of data has become a great challenge to the current storage system. The traditional centralized storage method can no longer meet the requirement of data storage, so distributed file systems for large-scale data storage, such as Google File system / GFSU / Hadoop File system HDFSU / PVFS Luster, etc., have good extensibility and fault tolerance. It can meet the demand of massive data storage. However, in many applications, it is necessary to support the storage of large files as well as large files. Although distributed file systems such as GFSU HDFS can satisfy the efficient storage of large files, the efficiency of storing large numbers of small files is very low. In order to solve this problem, many methods have been put forward by industry and academic circles. However, there are many problems such as low performance, low reliability of system and low efficient storage of small file metadata. Aiming at these challenges, this paper designs and implements a large amount of small file storage system based on HDFS. The main idea of this system is that, under the existing directory tree structure of HDFS, a small file in a folder is designed. Packaged into a large file for storage, this file is called a small file data file. At the same time, the index of small files is generated, and the location of small files in the corresponding data files is recorded. This paper designs and implements a large amount of small file storage system based on HDFS, which is an extensible, highly fault-tolerant and distributed large size small file storage cluster system. In this paper, we propose a small file aggregation storage technology to realize distributed data storage and fault tolerance by storing small file data in HDFS data file, at the same time, we propose a small file distributed index management technology to distribute the index to each data node management. It solves the problem that the single metadata node becomes the bottleneck in storing the large amount of small files, and the fault-tolerant mechanism of the index of the mass small file storage system can reduce the risk of small file loss by fault-tolerant of the index. By creating multiple data files in a single directory, the problem of accessing small files in the same directory is solved. On the basis of the above, the system caches the information of small file index position and data file flow, which is commonly used by users in the client side, and improves the efficiency of file access of the system. The experiment shows that the system has delayed reading and writing of small files. Throughput is much higher than native HDFS without small file support. Moreover, the system can effectively solve the problem that the large amount of metadata stored in small files is too large, and the reliability of the system is improved by index fault-tolerant mechanism.
【学位授予单位】：国防科学技术大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【参考文献】