一种基于MongoDB和HDFS的大规模遥感数据存储系统的设计与实现

发布时间：2018-03-04 14:20

本文选题：遥感数据　切入点：元数据　出处：《浙江大学》2013年硕士论文　论文类型：学位论文

【摘要】：遥感数据有多标准、多类型、多尺度、多级别、海量以及分布式存储的特征。随着遥感技术和信息处理技术的不断进步,不同类型、不同级别的遥感数据不断涌现,社会各领域对遥感观测数据的需求越来越大,我国各部门和科研机构都建立了针对各行业不同资源类型、彼此异构的遥感影像库,形成了一个分布式的、异构的、跨部门的、跨地域、资源类型多样的遥感数据库群,大大制约了各部门间遥感数据的共享和应用。为了满足对遥感数据的管理及共享需求,我们实现了一个基于MongoDB和HDFS的大规模遥感数据存储系统,本文介绍了系统的详细设计与实现,并重点介绍了异构遥感元数据集成以及海量遥感数据高效存储两项关键技术。针对遥感元数据多源、异构、海量等特点本文提出了一种基于映射模板的异构遥感元数据集成方法,可以实现异构遥感元数据的格式化统一及高效存储。并具有支持元数据动态扩展的能力,可以解析不断涌现的新类型新格式遥感元数据,解决了以往的元数据管理系统扩展性兼容性差且不利于数据共享的问题。遥感元数据特点是异构、只读、小文件、海量。而遥感影像数据不但具有只读、海量的特点,而且单个遥感影像数据文多为GB数量级的大文件,而且多为冷数据访问频次少。系统的存储层通过采用遥感元数据和遥感影像数据分离存放的策略,并针对两种数据的特点进行了优化。针对遥感元数据采用了基于MongoDB的存储架构,系统不但能够提供高效的数据存储,而且具有高可靠性、高扩展性的特点。针对遥感影像数据系统采用基于HDFS的分布式文件存储架构,而且为了提高存储资源利用率优化了HDFS的多副本存储策略,提供了基于文件访问频次的混合存储策略,在保证数据可靠性和访问速度的前提下提高系统存储资源利用率。
[Abstract]:Remote sensing data has the characteristics of multi-standard, multi-type, multi-scale, multi-level, massive and distributed storage. With the development of remote sensing technology and information processing technology, different types and different levels of remote sensing data are emerging. There is a growing demand for remote sensing observation data in various fields of society. Various departments and scientific research institutions in China have established a remote sensing image database for different types of resources in various industries, which is heterogeneous to each other, forming a distributed, heterogeneous, cross-sectoral remote sensing image database. In order to meet the requirement of remote sensing data management and sharing, remote sensing data sharing and application among different departments are greatly restricted by remote sensing database groups with diverse resource types across regions. We implement a large-scale remote sensing data storage system based on MongoDB and HDFS. This paper introduces the detailed design and implementation of the system, and focuses on two key technologies: heterogeneous remote sensing metadata integration and efficient storage of massive remote sensing data. According to the characteristics of multi-source, heterogeneity and magnanimity of remote sensing metadata, this paper presents an integration method of heterogeneous remote sensing metadata based on mapping template. It can realize the unified and efficient storage of heterogeneous remote sensing metadata, support the dynamic expansion of metadata, and parse the emerging new types and formats of remote sensing metadata. It solves the problem of poor extensibility compatibility and bad data sharing of metadata management systems in the past. Remote sensing metadata is characterized by heterogeneity, read-only, small file, mass, and remote sensing image data not only has read-only, magnanimous characteristics, but also single remote sensing image data text is mostly GB large file, The storage layer of the system adopts the strategy of separating remote sensing metadata from remote sensing image data. Aiming at the characteristics of the two kinds of data, the storage architecture based on MongoDB is adopted for remote sensing metadata. The system not only can provide efficient data storage, but also has high reliability. Aiming at remote sensing image data system, the distributed file storage architecture based on HDFS is adopted. In order to improve the utilization of storage resources, the multi-copy storage strategy of HDFS is optimized. A hybrid storage strategy based on the frequency of file access is provided to improve the utilization of storage resources on the premise of data reliability and access speed.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP751;TP333

【参考文献】