Hadoop架构下海量空间数据存储与管理

发布时间：2017-12-30 19:09

本文关键词：Hadoop架构下海量空间数据存储与管理　出处：《武汉大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着地理信息产业的飞速发展,作为GIS血液的地理空间数据正在随着指数级的速率增长,使得空间数据的检索、计算及分析等操作的难度不断加大,并且其应用到GIS的工作领域范围也在不断扩张,空间数据精度需求不断增大,导致海量空间数据的存储管理难度一再提升,迫切需要新的方法和技术来解决该问题。2005年开始Hadoop分布式系统开源技术迅速发展,其主要的两项技术HDFS和MapReduce可以为地理空间数据的分布式存储及并行计算处理提供技术支撑,为解决上述问题提供了一条新的思路。本文深入研究了地理空间数据的常见数据存储模式和数据存储结构,针对Hadoop分布式开源平台中的分布式文件系统HDFS存储需求设计了非结构化空间数据存储模式,以便海量空间数据进行分布式存储,分散计算机硬盘及处理器的压力。同时为将不同来源、不同格式、不同数据结构的地理空间数据统一存储在HDFS中,设计了统一化的数据转换接口。当前,空间数据索引的研究大都是针对单机建立的索引,很少有分布式存储空间索引研究。存储在HDFS中的空间数据完全无序,并且分布散乱,对空间数据的检索需要在集群中的每一个节点进行遍历,才能检索到用户所需要的空间数据。因此本文针对几种不同的数据划分算法深入研究,找出了适合海量空间数据索引管理的STR树索引,建立了数据划分—局部索引—全局索引的STR树空间索引机制。通过MapReduce并行处理框架对索引进行分布式计算,并在最后Reduce阶段将结果进行归并以返回给用户,优化了数据检索技术,大大提高了数据检索的效率。
[Abstract]:With the rapid development of geographic information industry, the geospatial data as GIS blood is increasing with the exponential rate, which makes the retrieval, calculation and analysis of spatial data increasingly difficult. And its application to the scope of GIS work is also expanding, spatial data accuracy requirements continue to increase, resulting in massive spatial data storage and management difficulties. New methods and technologies are urgently needed to solve this problem. In 2005, Hadoop distributed system open source technology developed rapidly. The two main technologies, HDFS and MapReduce, can provide technical support for distributed storage and parallel computing of geospatial data. It provides a new way to solve the above problems. This paper deeply studies the common data storage mode and data storage structure of geospatial data. An unstructured spatial data storage model is designed to meet the storage requirements of distributed file system (HDFS) in the Hadoop distributed open source platform in order to store large amounts of spatial data in distributed storage. Distributed computer hard disk and processor pressure. At the same time for different sources, different formats, different data structure of geospatial data stored in HDFS, designed a unified data conversion interface. Most of the research on spatial data index is based on single machine. There are few distributed storage spatial indexes. The spatial data stored in HDFS is completely disordered and scattered. Spatial data retrieval needs to traverse every node in the cluster in order to retrieve the spatial data required by users. Find out the STR tree index which is suitable for the massive spatial data index management. The STR tree spatial index mechanism of data partitioning, local index and global index is established, and the distributed computation of the index is carried out through the MapReduce parallel processing framework. In the final stage of Reduce, the result is merged to return to the user, the data retrieval technology is optimized, and the efficiency of data retrieval is greatly improved.
【学位授予单位】：武汉大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：P208

【参考文献】