云环境下图数据存储与并行聚类研究

发布时间：2018-04-02 09:10

本文选题：云存储　切入点：NoSQL　出处：《福州大学》2013年硕士论文

【摘要】：如何实现海量空间数据高效地存储管理和在线服务,同时基于这些数据挖掘用户感兴趣、有用的地学知识,成为地学领域日益关注的热点问题。目前空间数据云存储的研究大多集中在栅格数据存储和管理等方面,而针对矢量数据云存储与云服务领域研究较为零散,同时基于云存储的空间挖掘也缺乏系统的研究。本文根据矢量和栅格空间数据的不同特点,提出并实现了矢量栅格数据一体化的海量空间数据分布式云存储管理与访问服务方案,并基于云存储设计与实现了一个高效的并行谱聚类挖掘算法。主要研究内容和成果如下：1.在综述云计算、NoSQL数据库与图计算等相关领域的国内外研究现状的基础上,详细阐述了空间云存储与并行聚类的基础理论。分析了云计算模式的概念和特性以及NoSQL非关系数据库技术的起源、类型,分析了图数据库的数据模型、应用场景和传统空间存储技术的实现与局限,以及不同并行计算模型的原理、适用场合以及优缺点。2.根据矢量和栅格空间数据的不同特点,提出矢量栅格数据一体化的海量空间数据分布式云存储管理与访问服务方案。在三层式空间数据云存储架构基础上,给出基于NoSQL数据库技术的栅格和矢量数据云存储的实现策略与方法,并开展了通用数据访问接口的设计。采用分布式文件系统HDFS存储栅格数据,并使用列族数据库HBase对其建立分布式空间索引,而采用满足ACID约束的分布式图数据库Neo4J来存储矢量数据,并使用R树建立空间索引。3.探讨了基于图论的空间聚类算法——谱聚类算法原理,通过对比分析算法中不同的子图划分方法,采用DiDiC分割算法应用于并行图数据空间聚类中。同时结合并行计算框架MapReduce,提出了空间谱聚类算法流程及其基于MapReduce并行化的实现方法。4.开展了图数据的存储以及并行聚类的两个方面的试验测试。在图数据存储方面,开展了GeoDAC与PostGIS在矢量数据读写访问性能方面的对比测试。结果表明,虽然GeoDAC没有获得写入性能的加速作用,但其具有PostGIS无法比拟的强大读取性能。在图挖掘方面,开展了并行谱聚类算法与单机版谱聚类算法在图数据的挖掘效率对比测试,证明算法并行化具有显著提升图数据挖掘算法的性能。
[Abstract]:How to realize the efficient storage, management and online service of massive spatial data, and how to mine the interesting and useful geoscience knowledge of users based on these data, At present, the research of spatial data cloud storage is mostly focused on raster data storage and management, but the research on vector data cloud storage and cloud service is scattered. At the same time, space mining based on cloud storage is also lack of systematic research. According to the different characteristics of vector and raster spatial data, A distributed cloud storage management and access service scheme for mass spatial data is proposed and implemented, which integrates vector raster data. An efficient parallel spectral clustering algorithm based on cloud storage is designed and implemented. The main research contents and results are as follows: 1. On the basis of summarizing the research status of cloud computing NoSQL database and graph computing at home and abroad. The basic theory of spatial cloud storage and parallel clustering is described in detail, the concept and characteristics of cloud computing mode and the origin and type of NoSQL non-relational database technology are analyzed, and the data model of graph database is analyzed. The realization and limitation of application scene and traditional space storage technology, as well as the principle of different parallel computing models, the applicable situation and the advantages and disadvantages. 2.According to the different characteristics of vector and raster spatial data, This paper proposes a distributed cloud storage management and access service scheme for massive spatial data based on the integration of vector raster data, which is based on the three-layer spatial data cloud storage architecture. The implementation strategy and method of grid and vector data cloud storage based on NoSQL database technology are presented, and the design of general data access interface is carried out. The distributed file system HDFS is used to store grid data. The column family database HBase is used to build the distributed spatial index, and the distributed graph database Neo4J, which meets the ACID constraints, is used to store vector data. Using R-tree to build spatial index. 3. The principle of spectral clustering algorithm based on graph theory is discussed, and the different subgraph partition methods in the algorithm are compared and analyzed. The DiDiC segmentation algorithm is applied to the spatial clustering of parallel graph data. At the same time, combining with the parallel computing framework MapReduce, the flow of spatial spectral clustering algorithm and its realization method based on MapReduce parallelization are proposed. Test tests on two aspects of clustering. In the case of graph data storage, A comparative test between GeoDAC and PostGIS in vector data read and write access performance is carried out. The results show that although GeoDAC does not achieve the acceleration of write performance, it has a powerful read performance that cannot be compared with PostGIS. The parallel spectral clustering algorithm and the single-machine version spectral clustering algorithm are compared in graph data mining efficiency. It is proved that the parallelization of the algorithm can significantly improve the performance of graph data mining algorithm.
【学位授予单位】：福州大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：P208

【相似文献】