基于HBase的矢量空间数据存取关键技术研究

发布时间：2018-07-23 14:36

【摘要】：随着信息技术和空间信息获取技术的发展、全球信息化推进和GIS(地理信息系统)的广泛应用,空间数据高速增长。面对日益增长的海量空间数据,传统的空间数据管理方案面临高并发读写以及扩展性等等瓶颈。而云计算高扩展的存储能力以及强大的计算能力则可以满足海量数据存储、大数据并行处理、高并发检索等方面的需求。鉴于云计算技术的诸多优点,论文针对如何利用云计算技术实现对海量矢量空间数据的存取展开研究。重点对云平台下矢量空间数据的存储模型、空间索引构建、数据组织方案、数据的导入、空间查询策略以及在HBase上进行属性SQL查询等进行了研究与设计。论文围绕以下几个方面开展工作:(1)矢量空间数据云存储与检索研究背景介绍及相关理论技术分析。论文阐述了海量空间数据存储云存取的研究背景以及意义;分析了当前国内外云计算概况及空间数据云存取的研究现状以及当前研究的不足;结合Map Reduce并行计算框架特性深入分析了Map Reduce的矢量空间数据并行处理可行性,并探讨分布式数据库HBase以及SQL On Hadoop相关云计算技术存储和管理海量矢量空间数据的优势。(2)构建了基于HBase的矢量空间数据存储模型以及No SQL模型与关系模型一体化的矢量空间数据的管理方案。针对矢量空间数据的特点,结合HBase数据模型,设计了矢量空间数据存储模型,并采用四叉树层次剖分技术设计了多级格网索引;结合空间信息多级格网编码和Hilbert空间填充曲线的聚类特性,设计了符合HBase数据库Row Key存储规则的矢量空间数据标识编码;根据HBase数据库存储规则以及Phoenix操作结构化数据特性,提出并设计了No SQL模型与关系模型一体化的矢量空间数据的管理方案。(3)设计了矢量空间数据入库以及并行构建空间索引策略。结合Map Reduce并行处理特性讨论并设计了单机导入和基于Map Reduce并行处理矢量空间数据的入库方案以及基于Map Reduce设计了并行构建空间索引方案。(4)根据多级网格索引策略设计了空间查询策略。根据不同空间查询算子、多级网格索引特点以及HBase扫描查询数据特性,设计并实现了空间查询算子优化策略、合并网格编码优化查询策略以及限制扫描列簇优化数据过滤策略等三种空间查询优化策略。最后设计并实现了基于HBase的矢量空间数据存取原型系统,实现了网格索引以及多级网格索引,通过网格索引与多级网格索引空间查询效率对比实验,验证了多级网格索引的有效性。并基于多级网格索引,验证了空间查询算子优化策略、合并网格编码优化查询策略以及限制扫描列簇优化数据过滤策略等三种空间查询优化策略的有效性。
[Abstract]:With the development of information technology and spatial information acquisition technology, the development of global information technology and the wide application of GIS (Geographic Information system), spatial data is growing rapidly. In the face of the increasing amount of spatial data, the traditional spatial data management scheme faces the bottleneck of high concurrent reading and writing and expansibility. Cloud computing can meet the needs of massive data storage, big data parallel processing, high concurrent retrieval and so on. In view of the many advantages of cloud computing technology, this paper focuses on how to use cloud computing technology to access mass vector space data. This paper focuses on the research and design of vector spatial data storage model, spatial index construction, data organization scheme, data import, spatial query strategy and attribute SQL query on HBase. This paper focuses on the following aspects: (1) introduction to the research background of vector spatial data cloud storage and retrieval and analysis of related theory and technology. This paper describes the research background and significance of cloud access for massive spatial data storage, analyzes the general situation of cloud computing at home and abroad, the research status quo of cloud access of spatial data and the shortcomings of current research. Combined with the characteristics of Map Reduce parallel computing framework, the feasibility of vector spatial data parallel processing in Map Reduce is analyzed. The advantages of distributed database HBase and SQL On Hadoop related cloud computing technology in storing and managing massive vector spatial data are discussed. (2) the vector spatial data storage model based on HBase and the integration of No SQL model and relational model are constructed. Vector spatial data management scheme. According to the characteristics of vector spatial data, combined with the HBase data model, the vector spatial data storage model is designed, and the multilevel grid index is designed by using the quadtree hierarchical partition technology. Combined with the clustering characteristics of spatial information multilevel grid coding and Hilbert space filling curve, the vector spatial data identification coding is designed according to HBase database Row Key storage rules, according to HBase database storage rules and Phoenix operation structured data characteristics. This paper proposes and designs a vector spatial data management scheme which integrates No SQL model and relational model. (3) A vector spatial data storage strategy and a parallel spatial index strategy are designed. Combined with the characteristics of Map Reduce parallel processing, this paper discusses and designs the input scheme of single machine importing vector spatial data and Map Reduce parallel processing vector spatial data, and designs a parallel spatial index scheme based on Map Reduce. (4) according to the multi-level grid index strategy, we design a parallel spatial index scheme. Spatial query strategy is designed. According to the characteristics of different spatial query operators, multilevel grid index and HBase scanning query data, the optimization strategy of spatial query operator is designed and implemented. There are three spatial query optimization strategies: merging trellis coding optimizing query strategy and restricting scanning column cluster optimizing data filtering strategy. Finally, the prototype system of vector spatial data access based on HBase is designed and implemented. The grid index and multilevel grid index are implemented. The efficiency of spatial query between grid index and multilevel grid index is compared. The validity of multilevel grid index is verified. Based on the multilevel grid index, the effectiveness of three spatial query optimization strategies, namely spatial query operator optimization strategy, combined grid coding optimization query strategy and restricted scan column cluster optimization data filtering strategy, is verified.
【学位授予单位】：江西理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：P208

【参考文献】