基于MongoDB的海量大中小文件存储系统的研究与应用

发布时间：2018-02-20 06:44

本文关键词： 海量大中小文件存储模型数据接口均衡算法　出处：《中国地质大学(北京)》2016年硕士论文　论文类型：学位论文

【摘要】：随着互联网的日益发展和社交网络的日益盛行,异构网络数据不断增加。海量小文件的存储优化已经成为了当今海量数据存储技术的一个重要研究方向。对于HDFS、TFS等分布式文件系统,在处理海量小文件时不具有一般性。然而,随着NoSQL技术的逐渐成熟,它具有的分布式系统的优势、简单灵活的特点,也使它成为了解决海量小文件存储的可能。子午工程数据中心负责处理来自全国各地探测设备所产生的空间科学数据文件,随着空间探测数据的不断增加,截至2015年底,子午工程数据中心累计汇集科学数据980.18万个,文件总大小约3.45TB,而且其中90%是100k以下的小文件,其余的是少量中大文件。目前子午工程采用传统分布式文件系统来存储科学数据,在处理众多小文件时,会导致磁盘I/O过高,数据备份时间过长,数据存储效率较低。针对子午工程数据文件特点,充分分析了当今主流海量数据存储方案的优缺点,在MongoDB的基础上,本文提出了一种ZW-Mongo存储模型。该存储模型主要包括三方面设计:(1)利用MongoDB的BSON数据结构特性,直接处理小文件存储,提高小文件存储效率;(2)针对大文件分块存储,并构建元信息集合和块数据集合;(3)采用历史版本和软删除的方式,提高文件利用率。ZW-Mongo存储模型提高了小文件的存储和访问效率,有效地降低了文件的管理成本。通过分析MongoDB数据均衡策略的缺点,本文提出了基于一致性哈希的数据均衡策略,并构建了基于一致性哈希算法的文件存储过程。本文基于ZW-Mongo存储模型,设计并研发了一套REST风格的数据访问接口,同时实现了数据均衡算法的访问接口,便于数据节点的添加和移除。最后,通过ZW-Mongo存储模型数据接口与传统分布式文件系统对比测试表明,ZW-Mongo存储模型在数据读取、查询、备份等方面均优于传统存储模式,在数据写入方面两者基本相似,同时通过添加虚拟节点的数据均衡测试表明,添加虚拟节点的数量可以促进数据节点之间的均衡分布。ZW-Mongo存储模型已实际应用于子午工程数据中心的数据存储子系统中,应用效果良好。
[Abstract]:With the development of the Internet and the growing popularity of social networks, The storage optimization of large amount of small files has become an important research direction of mass data storage technology. For distributed file systems such as HDFS / TFS, it is not general when dealing with large amounts of small files. With the maturity of NoSQL technology, it has the advantages of distributed system, simple and flexible. It also makes it possible to store large amounts of small files. Meridian Engineering data Center is responsible for processing space science data files generated by exploration equipment from all over the country. As space exploration data continues to increase, as of end of 2015, Meridian Engineering data Center accumulates 980.18 million scientific data, and the total file size is about 3.45 TB.And 90% of them are small files below 100k, and the rest are a small number of medium and large files. At present, the Meridian Project uses traditional distributed file systems to store scientific data. When dealing with many small files, the disk I / O is too high, data backup time is too long, and data storage efficiency is low. According to the characteristics of meridian engineering data files, the advantages and disadvantages of current mainstream massive data storage schemes are analyzed. On the basis of MongoDB, this paper proposes a ZW-Mongo storage model. The storage model includes three aspects: design: 1) using the BSON data structure of MongoDB to deal with small file storage directly and improve the efficiency of small file storage. Using the historical version and soft delete method, we can improve the file utilization. ZW-Mongo storage model improves the storage and access efficiency of small files. By analyzing the shortcomings of MongoDB data equalization strategy, this paper proposes a data equalization strategy based on consistency hash. Based on the ZW-Mongo storage model, a set of REST style data access interface is designed and developed in this paper. At the same time, the data equalization algorithm access interface is realized. Finally, by comparing the ZW-Mongo storage model data interface with the traditional distributed file system, it shows that the ZW-Mongo storage model is superior to the traditional storage mode in data reading, query, backup and so on. In the aspect of data writing, they are basically similar. At the same time, the data equalization test of adding virtual nodes shows that, The addition of the number of virtual nodes can promote the balanced distribution of data nodes. ZW-Mongo storage model has been applied to the data storage subsystem of meridian engineering data center and the application effect is good.
【学位授予单位】：中国地质大学(北京)
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP333

【参考文献】