海量GNSS小文件云存储优化方法研究

发布时间：2018-11-12 16:56

【摘要】：GNSS数据量呈指数级趋势增长,Hadoop分布式文件系统(HDFS)解决了海量GNSS数据存储瓶颈的难题,却面临内存占用多、文件相关性差和缺乏优化机制的问题。针对HDFS处理海量GNSS小文件效率不高的问题,结合GNSS数据类型、特点以及存储过程,提出了一种新的GNSS小文件云存储方法,优化了GNSS小文件的写入、读取、添加和删除策略。该方法分别按观测文件和解算成果的类型进行合并,对合并后的文件构建压缩Trie树索引,索引切分后,根据匹配算法分布式地存储索引块。实验采用国际GNSS服务(IGS)28d的数据和产品进行云存储优化。结果表明,该方法降低了各节点内存消耗,提高了海量GNSS小文件写入、读取和删除的效率,实现了对海量GNSS小文件的高效云存储。
[Abstract]:The amount of GNSS data increases exponentially. Hadoop distributed file system (HDFS) solves the bottleneck problem of massive GNSS data storage, but it faces the problems of large memory consumption, poor file correlation and lack of optimization mechanism. In order to solve the problem that HDFS is not efficient in handling large amounts of GNSS small files, a new method of cloud storage of GNSS small files is proposed, which optimizes the strategy of writing, reading, adding and deleting GNSS small files, combined with the data types, characteristics and stored procedures of GNSS. In this method, the index of compressed Trie tree is constructed according to the type of the observed file and the result of computation, and the index block is stored distributed according to the matching algorithm after the index is segmented. The experiment adopts the data and product of international GNSS service (IGS) 28d for cloud storage optimization. The results show that this method reduces the memory consumption of each node, improves the efficiency of writing, reading and deleting massive GNSS small files, and realizes the efficient cloud storage of massive GNSS small files.
【作者单位】：地理信息工程国家重点实验室;信息工程大学地理空间信息学院;
【基金】：国家自然科学基金(41674019) 国家重点研发计划(2016YFB0501701) 地理信息工程国家重点实验室开放基金(SKLGIE2016-M-1-2)~~
【分类号】：P228.4

【参考文献】