基于Hadoop的云GIS若干关键技术研究
发布时间：2018-11-13 08:48
【摘要】：云计算是互联网计算发展到一定阶段的产物,是并行计算、网格计算等多种新型计算方式演进的最新结果。云计算无限扩展的存储技术可以满足快速增长的空间数据对存储空间的需求,强大的计算能力可为空间信息的检索、处理、分析等提供高速的服务保证。本文针对GIS当前所面临的海量数据存储、处理、分析与持续服务等问题,结合GIS和云计算的特点,将开源Hadoop云计算平台应用到空间信息服务领域,研究利用Hadoop云计算平台提供的分布式存储能力和并行计算能力,构建基于Hadoop的GIS应用,并对其中的一些关键技术进行研究。本文主要工作如下:(1)在分析商业云GIS体系结构的基础上,设计了基于Hadoop的云GIS体系结构。体系结构包括物理设备层、平台层、软件层、应用层等4层,以及横跨多个层次的用户管理、服务管理、资源管理、监控系统、容灾备份、运营管理等服务。设计了基于Hadoop的云GIS部署模式。整个基于Hadoop的云GIS系统由平台管理门户、GIS Web服务器集群及多个Hadoop集群组成。分析了体系结构特点,为后面的研究内容奠定了基础。(2)本文在空间信息格网单元和OGC简单要素规范基础上,结合矢量数据的特点,利用格网单元ID的唯一性、多尺度性及索引性,提出了一种以格网单元为存储单位的矢量数据存储方案;结合矢量要素的定性属性数据,设计了矢量数据的存储格式“GWKT(Grid Well-know Text)”;为了达到矢量要素标识全球唯一,本文基于格网单元和Hilbert曲线的Base16编码,结合HBase数据库的特点,设计了矢量要素标识的编码,并实现了编码的生成算法;研究实现了基于单调链的矢量要素分割与合并算法,可有效的分割和合并线状和面状要素;在HBase基础上,扩展了HBase的数据类型及其过滤器,实现了属性数据的快速查询。(3)针对海量空间数据处理能力不足问题,设计了基于HDFS的矢量数据存储格式,实现了基于MapReduce的矢量数据分割入库并行处理模型;在MapReduce数据过滤器的基础上,设计了适合基于格网单元的空间数据并行计算模型,并以矢量数据缓冲区分析作为实例进行了验证;设计实现了基于MapReduce的kNN空间数据查询算法;分析了基于MapReduce的空间数据并行计算效率。(4)在空间信息服务方面,本文在OGC标准服务基础上,对服务参数进行了扩展,设计了云GIS空间信息服务分层体系结构,实现了基于空间信息多级格网的WMS、WMTS、WFS和WPS等服务;设计实现了空间信息服务接口,以实现客户端与服务器端的完全解耦。(5)在前文研究基础上,设计并实现了基于Hadoop的云GIS原型系统,完成了海量栅格与矢量数据的高效存储与管理、空间数据并行计算以及基于Hadoop的空间信息服务等关键模块;并对相关模块做了性能测试,验证了本文提出的相关存储模型和计算模型的可行性、有效性以及高效性。
[Abstract]:Cloud computing is the product of the development of Internet computing to a certain stage. It is the latest result of the evolution of many new computing methods, such as parallel computing, grid computing and so on. The storage technology of infinite expansion of cloud computing can meet the demand for storage space of rapidly growing spatial data. Powerful computing power can provide high-speed service guarantee for spatial information retrieval, processing, analysis and so on. Aiming at the problems of massive data storage, processing, analysis and continuous service in GIS, combining the characteristics of GIS and cloud computing, the open source Hadoop cloud computing platform is applied to the field of spatial information service. The distributed storage and parallel computing capabilities provided by Hadoop cloud computing platform are used to construct GIS applications based on Hadoop, and some key technologies are studied. The main work of this paper is as follows: (1) based on the analysis of business cloud GIS architecture, a cloud GIS architecture based on Hadoop is designed. The architecture includes four layers: physical device layer, platform layer, software layer, application layer, and so on, as well as services such as user management, service management, resource management, monitoring system, disaster recovery backup, operation management and so on. A cloud GIS deployment model based on Hadoop is designed. The whole cloud GIS system based on Hadoop consists of platform management portal, GIS Web server cluster and multiple Hadoop clusters. The characteristics of the architecture are analyzed, which lays the foundation for the later research. (2) based on the specification of spatial information grid element and OGC simple element, combined with the characteristics of vector data, the uniqueness of grid element ID is used in this paper. In this paper, a vector data storage scheme with grid unit as storage unit is proposed, which is multiscale and indexed. Combining the qualitative attribute data of vector elements, the storage format "GWKT (Grid Well-know Text" of vector data is designed. In order to achieve the global uniqueness of vector element identification, based on the Base16 coding of grid element and Hilbert curve, combined with the characteristics of HBase database, the encoding of vector element identification is designed, and the coding algorithm is realized. The algorithm of vector element segmentation and merging based on monotone chain is studied, which can effectively segment and merge linear and plane elements. On the basis of HBase, the data type and its filter of HBase are extended, and the fast query of attribute data is realized. (3) aiming at the problem of insufficient processing ability of massive spatial data, the vector data storage format based on HDFS is designed. The parallel processing model of vector data segmentation and input database based on MapReduce is implemented. On the basis of MapReduce data filter, the parallel computing model of spatial data based on grid element is designed, and the vector data buffer analysis is used as an example to verify it, and the kNN spatial data query algorithm based on MapReduce is designed and implemented. The efficiency of parallel computing of spatial data based on MapReduce is analyzed. (4) in the aspect of spatial information service, the service parameters are extended on the basis of OGC standard service, and the hierarchical architecture of cloud GIS spatial information service is designed. The services such as WMS,WMTS,WFS and WPS based on spatial information multilevel grid are realized. The spatial information service interface is designed and implemented to realize the complete decoupling between the client and the server. (5) based on the previous research, a cloud GIS prototype system based on Hadoop is designed and implemented. The key modules, such as efficient storage and management of massive grid and vector data, parallel computing of spatial data and spatial information service based on Hadoop, are completed. The performance tests of the related modules are carried out to verify the feasibility, effectiveness and efficiency of the related storage model and the computing model proposed in this paper.
【学位授予单位】：解放军信息工程大学
【学位级别】：博士
【学位授予年份】：2013
【分类号】：P208
本文编号：2328605
[Abstract]:Cloud computing is the product of the development of Internet computing to a certain stage. It is the latest result of the evolution of many new computing methods, such as parallel computing, grid computing and so on. The storage technology of infinite expansion of cloud computing can meet the demand for storage space of rapidly growing spatial data. Powerful computing power can provide high-speed service guarantee for spatial information retrieval, processing, analysis and so on. Aiming at the problems of massive data storage, processing, analysis and continuous service in GIS, combining the characteristics of GIS and cloud computing, the open source Hadoop cloud computing platform is applied to the field of spatial information service. The distributed storage and parallel computing capabilities provided by Hadoop cloud computing platform are used to construct GIS applications based on Hadoop, and some key technologies are studied. The main work of this paper is as follows: (1) based on the analysis of business cloud GIS architecture, a cloud GIS architecture based on Hadoop is designed. The architecture includes four layers: physical device layer, platform layer, software layer, application layer, and so on, as well as services such as user management, service management, resource management, monitoring system, disaster recovery backup, operation management and so on. A cloud GIS deployment model based on Hadoop is designed. The whole cloud GIS system based on Hadoop consists of platform management portal, GIS Web server cluster and multiple Hadoop clusters. The characteristics of the architecture are analyzed, which lays the foundation for the later research. (2) based on the specification of spatial information grid element and OGC simple element, combined with the characteristics of vector data, the uniqueness of grid element ID is used in this paper. In this paper, a vector data storage scheme with grid unit as storage unit is proposed, which is multiscale and indexed. Combining the qualitative attribute data of vector elements, the storage format "GWKT (Grid Well-know Text" of vector data is designed. In order to achieve the global uniqueness of vector element identification, based on the Base16 coding of grid element and Hilbert curve, combined with the characteristics of HBase database, the encoding of vector element identification is designed, and the coding algorithm is realized. The algorithm of vector element segmentation and merging based on monotone chain is studied, which can effectively segment and merge linear and plane elements. On the basis of HBase, the data type and its filter of HBase are extended, and the fast query of attribute data is realized. (3) aiming at the problem of insufficient processing ability of massive spatial data, the vector data storage format based on HDFS is designed. The parallel processing model of vector data segmentation and input database based on MapReduce is implemented. On the basis of MapReduce data filter, the parallel computing model of spatial data based on grid element is designed, and the vector data buffer analysis is used as an example to verify it, and the kNN spatial data query algorithm based on MapReduce is designed and implemented. The efficiency of parallel computing of spatial data based on MapReduce is analyzed. (4) in the aspect of spatial information service, the service parameters are extended on the basis of OGC standard service, and the hierarchical architecture of cloud GIS spatial information service is designed. The services such as WMS,WMTS,WFS and WPS based on spatial information multilevel grid are realized. The spatial information service interface is designed and implemented to realize the complete decoupling between the client and the server. (5) based on the previous research, a cloud GIS prototype system based on Hadoop is designed and implemented. The key modules, such as efficient storage and management of massive grid and vector data, parallel computing of spatial data and spatial information service based on Hadoop, are completed. The performance tests of the related modules are carried out to verify the feasibility, effectiveness and efficiency of the related storage model and the computing model proposed in this paper.
【学位授予单位】：解放军信息工程大学
【学位级别】：博士
【学位授予年份】：2013
【分类号】：P208
【参考文献】
相关期刊论文 前10条
1 刘义;景宁;陈荦;熊伟;;MapReduce框架下基于R-树的k-近邻连接算法[J];软件学报;2013年08期
2 陈崇成;林剑峰;吴小竹;巫建伟;连惠群;;基于NoSQL的海量空间数据云存储与服务方法[J];地球信息科学学报;2013年02期
3 夏英;杨选伦;;云环境中基于金字塔模型的影像数据存储方法[J];重庆邮电大学学报(自然科学版);2012年06期
4 李德仁;姚远;邵振峰;;智慧地球时代测绘地理信息学的新使命[J];测绘科学;2012年06期
5 林德根;梁勤欧;;云GIS的内涵与研究进展[J];地理科学进展;2012年11期
6 范协裕;任应超;杨崇俊;唐建智;易雄鹰;;基于集群技术的可伸缩云GIS服务平台研究[J];计算机应用研究;2012年10期
7 张广弟;汪秀兵;胡亚磊;;基于hadoop的DEM格网建立研究[J];科技视界;2012年28期
8 苟丽美;朱美正;李艳明;;RESTful风格地图瓦片服务的研究[J];计算机工程与设计;2012年09期
9 王旭东;张福浩;张丽萍;;基于地形数据的瓦片金字塔构建与组织[J];测绘与空间地理信息;2012年06期
10 张鹏飞;陈荣国;谢炯;张明波;;空间数据库的WFS长事务管理方案研究[J];测绘科学技术学报;2012年03期
相关博士学位论文 前2条
1 康俊锋;云计算环境下高分辨率遥感影像存储与高效管理技术研究[D];浙江大学;2011年
2 赵春宇;高性能并行GIS中矢量空间数据存取与处理关键技术研究[D];武汉大学;2006年
相关硕士学位论文 前3条
1 杨柳;基于云计算的GIS应用模式研究[D];河南大学;2011年
2 霍树民;基于Hadoop的海量影像数据管理关键技术研究[D];国防科学技术大学;2010年
3 岳昌华;空间数据库K最接近对查询算法的研究[D];大连海事大学;2010年
，本文编号：2328605
本文链接：https://www.wllwen.com/kejilunwen/dizhicehuilunwen/2328605.html
