面向并行数字地形分析的DEM数据云存储方法研究
发布时间：2018-10-20 08:45
【摘要】：人类对地观测技术的发展使得大范围高精度地形数字高程模型(DEM)数据的数据量呈爆炸式增长。基于单一计算机的传统数字地形分析方法由于存储与计算性能有限,无法满足地学研究和生产应用的需求。面向新型硬件架构的并行数字地形分析理论和方法的研究已经成为地学界的研究热点之一。然而,目前的研究着重于考虑地形分析算法的并行,很少涉及面向并行数字地形分析的海量DEM数据以及数字地形分析结果的管理问题,已有的DEM数据管理模式也无法满足并行数字地形分析的需求。分布式计算平台Hadoop被认为是海量数据处理和分析的利器。如何利用Hadoop来管理急剧增加的DEM数据,以满足并行数字地形分析的需求,是值得研究的课题,本文即围绕此课题展开研究,主要内容包括以下三个方面： 1)面向并行数字地形分析的DEM数据组织方式 在深入分析并行数字地形分析的特点及其对DEM数据组织和管理需求的基础上,详细介绍了DEM金字塔结构及其构建方法和数学模型,并基于此结构设计了DEM及其增量数据的分布式存储模型,可实现海量DEM数据及其并行数字地形分析结果的有效存储。 21面向并行数字地形分析的DEM数据管理方法 DEM管理方法包括空间索引机制、数据压缩方法、系统容错机制和并发访问策略等。本文结合HBase数据库的存储模式,构建了三级空间索引,并提出了基于内容的空间索引方法,使系统同时支持基于特征查询和基于内容查询。在数据压缩方面,设计了高程增量游程编码压缩算法,并对算法的适用性进行了实验验证。此外,深入研究了Hadoop的容错方案和高并发访问策略,并结合DEM数据的特点,给出了配置参数的建议值以及部分实现代码。 3)DEM云存储原型系统的设计与实现 通过分析DEM云存储系统的设计需求,给出了系统的详细设计方案。并以三台PC机搭建Hadoop集群模拟DEM云存储环境,在该集群环境上开发了DEM云存储原型系统。该系统支持四种查询方式：①基于文件名查询；②基于范围和分辨率查询；③基于范围和格网数查询；④基于内容查询。以全球SRTM3数据为实验数据对系统进行验证。实验结果表明,数据检索的结果是完全正确的,在现有的集群环境条件下,数据检索的时间效率是令人满意的。
[Abstract]:With the development of human earth observation technology, the data volume of (DEM) data of large range and high precision terrain digital elevation model increases explosively. The traditional digital terrain analysis method based on a single computer is unable to meet the requirements of geoscience research and production applications because of its limited storage and computing performance. The research of parallel digital terrain analysis theory and method for new hardware architecture has become one of the research hotspots in geoscience. However, the current research focuses on considering the parallelism of terrain analysis algorithms, and rarely involves the massive DEM data for parallel digital terrain analysis and the management of digital terrain analysis results. The existing DEM data management model can not meet the needs of parallel digital terrain analysis. Hadoop, a distributed computing platform, is considered to be a powerful tool for mass data processing and analysis. How to use Hadoop to manage the rapidly increasing DEM data to meet the needs of parallel digital terrain analysis is a topic worthy of study. The main contents include the following three aspects: 1) the DEM data organization for parallel digital terrain analysis is based on the in-depth analysis of the characteristics of parallel digital terrain analysis and the requirements for DEM data organization and management. This paper introduces the pyramid structure of DEM, its construction method and mathematical model in detail, and designs the distributed storage model of DEM and its incremental data based on this structure. The efficient storage of massive DEM data and its parallel digital terrain analysis results can be realized. 21 the DEM data management method for parallel digital terrain analysis includes spatial index mechanism, data compression method, DEM management method. System fault tolerant mechanism and concurrent access strategy. Combined with the storage mode of HBase database, this paper constructs a three-level spatial index, and proposes a content-based spatial index method, which enables the system to support both feature-based query and content-based query. In the aspect of data compression, the algorithm of height increment run-length coding compression is designed, and the applicability of the algorithm is verified by experiments. In addition, the fault-tolerant scheme and high concurrency access strategy of Hadoop are deeply studied, and the characteristics of DEM data are combined. The design and implementation of DEM cloud storage prototype system are given. By analyzing the design requirements of DEM cloud storage system, the detailed design scheme of the system is given. The Hadoop cluster simulation DEM cloud storage environment is built with three PC computers, and the DEM cloud storage prototype system is developed in the cluster environment. The system supports four query modes: (1) file name query; (2) range and resolution based query; (3) range and mesh query; (4) content based query. The global SRTM3 data are used as experimental data to verify the system. The experimental results show that the results of data retrieval are correct and the time efficiency of data retrieval is satisfactory under the existing cluster environment.
【学位授予单位】：南京师范大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：P208
本文编号：2282643
[Abstract]:With the development of human earth observation technology, the data volume of (DEM) data of large range and high precision terrain digital elevation model increases explosively. The traditional digital terrain analysis method based on a single computer is unable to meet the requirements of geoscience research and production applications because of its limited storage and computing performance. The research of parallel digital terrain analysis theory and method for new hardware architecture has become one of the research hotspots in geoscience. However, the current research focuses on considering the parallelism of terrain analysis algorithms, and rarely involves the massive DEM data for parallel digital terrain analysis and the management of digital terrain analysis results. The existing DEM data management model can not meet the needs of parallel digital terrain analysis. Hadoop, a distributed computing platform, is considered to be a powerful tool for mass data processing and analysis. How to use Hadoop to manage the rapidly increasing DEM data to meet the needs of parallel digital terrain analysis is a topic worthy of study. The main contents include the following three aspects: 1) the DEM data organization for parallel digital terrain analysis is based on the in-depth analysis of the characteristics of parallel digital terrain analysis and the requirements for DEM data organization and management. This paper introduces the pyramid structure of DEM, its construction method and mathematical model in detail, and designs the distributed storage model of DEM and its incremental data based on this structure. The efficient storage of massive DEM data and its parallel digital terrain analysis results can be realized. 21 the DEM data management method for parallel digital terrain analysis includes spatial index mechanism, data compression method, DEM management method. System fault tolerant mechanism and concurrent access strategy. Combined with the storage mode of HBase database, this paper constructs a three-level spatial index, and proposes a content-based spatial index method, which enables the system to support both feature-based query and content-based query. In the aspect of data compression, the algorithm of height increment run-length coding compression is designed, and the applicability of the algorithm is verified by experiments. In addition, the fault-tolerant scheme and high concurrency access strategy of Hadoop are deeply studied, and the characteristics of DEM data are combined. The design and implementation of DEM cloud storage prototype system are given. By analyzing the design requirements of DEM cloud storage system, the detailed design scheme of the system is given. The Hadoop cluster simulation DEM cloud storage environment is built with three PC computers, and the DEM cloud storage prototype system is developed in the cluster environment. The system supports four query modes: (1) file name query; (2) range and resolution based query; (3) range and mesh query; (4) content based query. The global SRTM3 data are used as experimental data to verify the system. The experimental results show that the results of data retrieval are correct and the time efficiency of data retrieval is satisfactory under the existing cluster environment.
【学位授予单位】：南京师范大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：P208
【参考文献】
相关期刊论文 前6条
1 覃雄派;王会举;杜小勇;王珊;;大数据分析——RDBMS与MapReduce的竞争与共生[J];软件学报;2012年01期
2 方涛,李德仁,龚健雅,皮明红;GeoImageDB多分辨率无缝影像数据库系统的开发与实现[J];武汉测绘科技大学学报;1999年03期
3 陈仁喜;赵忠明;王殷行;;基于整型小波变换的DEM数据压缩[J];武汉大学学报(信息科学版);2006年04期
4 多雪松;张晶;高强;;基于Hadoop的海量数据管理系统[J];微计算机信息;2010年13期
5 郑晶晶;方金云;韩承德;;一种快速的DEM数据无损压缩算法[J];系统仿真学报;2010年10期
6 赵立成,王素娟,施进明;国家卫星气象中心信息共享体制研究与技术实现[J];应用气象学报;2002年05期
，本文编号：2282643
本文链接：https://www.wllwen.com/kejilunwen/dizhicehuilunwen/2282643.html
