基于NoSQL的海量航空物流数据分布式存储研究

发布时间：2018-03-28 22:24

本文选题：航空物流　切入点：海量小文件　出处：《中国民航大学》2017年硕士论文

【摘要】：航空物流信息系统是民航重要的系统,承载着频繁的数据交换、数据存储等任务,其数据大多以海量小文件形式存在。海量小文件由于种类繁多、数据规模较大,其存储问题已经成为工业界的难题。目前,大多企业使用分布式文件系统HDFS存储数据,然而HDFS是专门解决大文件存储问题设计的,对海量小文件的存储,面临着存储效率不高、内存占用量大、单点故障、磁盘空间利用率低等诸多问题。因此,研究航空物流海量小文件的分布式存储方法尤为重要。本文首先根据HDFS在存储海量小文件存在的诸多问题,分析了航空物流关键数据及其特征,提出了航空物流海量小文件优化存储方法,包括小文件的合并过程、预取机制、永久化方法,考虑文件之间的相关性,提高文件的存储和访问效率,考虑文件的时效性,提高磁盘空间的利用率。其次,通过组合优化存储方法,进而提出一种基于NoSQL的海量航空物流数据分布式多级存储架构,使用NoSQL内存式数据库Redis作缓存级,HDFS作数据的活跃级和永久级,数据的写入、数据的访问和数据永久化操作相互独立,降低了模块之间的耦合性;使Redis部署到HDFS的DataNode节点构成集群,减少HDFS中NameNode的单点内存压力,提升并发性能和索引检索速度,保持了元数据服务的可扩展性。最后,通过海量航空物流数据分布式存储实验与分析,该研究方法对海量小文件的存取效率有明显地提升,在顺序访问情况下,访问时间在160毫秒左右,同时减少了磁盘的空间占用,可节省79%左右的磁盘空间。
[Abstract]:Aviation logistics information system is an important system of civil aviation, carrying frequent data exchange, data storage and other tasks, its data mostly exist in the form of a large number of small files. At present, most enterprises use distributed file system (HDFS) to store data. However, HDFS is specially designed to solve the problem of large file storage. Therefore, it is very important to study the distributed storage method of large amount of small files in aeronautical logistics. Firstly, according to the problems existing in storing large amount of small files in HDFS, there are many problems in the storage of large numbers of small files in this paper, such as large amount of memory, single point failure, low utilization of disk space and so on. This paper analyzes the key data and its characteristics of aviation logistics, and puts forward the optimized storage method of large amount of small files in aviation logistics, including the merging process of small files, prefetching mechanism, permanence method and considering the correlation between files. Improve the efficiency of file storage and access, consider the timeliness of files, improve the utilization of disk space. Secondly, by combining and optimizing storage method, we propose a distributed multi-level storage architecture of massive aviation logistics data based on NoSQL. The NoSQL memory database Redis is used as the buffer level and Redis as the active and permanent level of data. The data writing, data accessing and data permanence operation are independent of each other, which reduces the coupling between modules. Redis is deployed to the DataNode nodes of HDFS to form a cluster, which reduces the single point memory pressure of NameNode in HDFS, improves concurrent performance and index retrieval speed, and maintains the extensibility of metadata services. Through the distributed storage experiment and analysis of massive aviation logistics data, this research method can obviously improve the access efficiency of large amount of small files. In the case of sequential access, the access time is about 160 milliseconds, and the space occupied by the disk is reduced at the same time. Can save about 79% of disk space.
【学位授予单位】：中国民航大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333;TP311.13

【参考文献】