一种基于HDFS的分布式多级R树空间索引研究

发布时间：2018-01-11 22:13

本文关键词：一种基于HDFS的分布式多级R树空间索引研究　出处：《中国测绘科学研究院》2016年硕士论文　论文类型：学位论文

【摘要】：随着测绘技术不断的发展,测绘地理信息部门拥有的地理数据飞速增长,而相应的空间数据存储与处理仍然较多使用传统的方法,己不能支持海量空间数据。随着计算机技术的不断发展,数据库技术以及文件组织方式发生了巨大的变化。如今云计算、分布式处理及并行网格计算技术的逐渐成熟,在各行各业有着广泛的应用。HDFS (Hadoop Distributed File System)以及MapReduce作为目前广泛应用的分布式存储与计算框架,支持利用计算机集群进行海量数据的存储与快速处理,这为提高海量矢量数据的存储与计算提供了新途径。因此面向日益增长的大数据集管理要求,将分布式技术引入空间数据存储与组织中是空间数据存储与处理方法研究的重要课题之一。本文通过将传统空间数据存储以及查询方法与HDFS分布式存储技术相结合,借助于可扩展的分布式文件系统HDFS存储空间数据,解决了海量空间数据的存储问题。由于传统的空间索引并不能很好的适用于分布式空间数据存储结构,因此本文设计了全局索引与局部索引相结合的分布式多级R树空间索引——DMLR(Distributed Multi-layer R Tree)。DMLR空间索引采用STR叶结点分割思想对空间数据进行分区,可以更好的应对数据分布不均衡的情况。利用Spark并行计算框架对DMLR索引进行并行构建,加快了DMLR索引的构建效率。此外通过将传统的空间数据查询方法与并行计算技术相结合,设计了基于DMLR空间索引的空间数据并行查询方法。包括并行范围聚集查询、并行k邻近查询、并行空间连接查询等方法。DMLR空间索引为分布式空间数据索引提供了一种新方法。通过空间数据并行查询实验也验证了DMLR索引在分布式环境下对海量空间数据管理的有效性。
[Abstract]:With the continuous development of surveying and mapping technology, the geographic data owned by the surveying and mapping geographic information department has increased rapidly, and the corresponding spatial data storage and processing still use the traditional methods. With the continuous development of computer technology, database technology and file organization have undergone tremendous changes. Now cloud computing. Distributed processing and parallel grid computing technology gradually mature. Hadoop Distributed File system is widely used in a variety of industries. And MapReduce is widely used as a distributed storage and computing framework. It provides a new way to improve the storage and computation of mass vector data by using computer cluster to store and process mass data quickly, so it faces the increasing requirement of large data set management. The introduction of distributed technology into spatial data storage and organization is one of the most important topics in the research of spatial data storage and processing. In this paper, traditional spatial data storage, query and HDFS distributed storage techniques are introduced. A combination of art. With the help of the extensible distributed file system (HDFS) to store spatial data, the storage problem of massive spatial data is solved, because the traditional spatial index is not suitable for distributed spatial data storage structure. Therefore, this paper designs a distributed multilevel R-tree spatial index combining global index and local index: DMLR distributed Multi-layer R tree). DMLR spatial index uses the idea of STR leaf node segmentation to partition spatial data. It can better deal with the uneven distribution of data. Using Spark parallel computing framework to build DMLR index in parallel. In addition, the traditional spatial data query method is combined with parallel computing technology. A spatial data parallel query method based on DMLR spatial index is designed, including parallel range aggregation query and parallel k-neighborhood query. Parallel spatial join query. DMLR spatial index provides a new method for distributed spatial data index. The experiment of spatial data parallel query also verifies the DMLR index to sea in distributed environment. The effectiveness of spatial data management.
【学位授予单位】：中国测绘科学研究院
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：P208

【相似文献】