协作式大规模地理栅格数据并行处理方法研究

发布时间：2018-07-14 18:40

【摘要】：以高分辨率遥感卫星为代表的新一代数据获取技术取得了较大进步,地理栅格数据在时空分辨率、数据类型、覆盖面积等方面不断提高,为地理应用提供了更多的数据信息,然而这些提高也带来数据量几何级增长,但同时也致使传统的遥感数据处理方法无法满足大规模地理栅格数据计算与分析需求。因此,研究面向大规模地理栅格数据的高性能计算方法与体系,进而为提高开发效率和解决复杂地学问题的能力具有十分重要的意义。为了解决大规模栅格地理数据并行处理所面临的难题,本文系统研究高性能计算架构下大规模地理栅格数据并行处理方法,引入MPI (Message Passing Interface)和MP (Multi processing)作为基础并行环境,研究栅格数据处理算法程序必备的功能和必须的流程,综合所有算法的共性,采用设计模式思想,构造符合面向对象程序基本原则的系统框架体系,构建一种协作式大规模栅格数据的并行处理框架(Cooperative Big Geographic Raster Data Parallel Processing Framework, CBGRDPPF),并结合地理栅格数据并行类型与复杂度等处理任务特点,探讨了在此框架下进行地学栅格数据处理任务的协同处理方法和技术,分析不同参数和环境对其运行速度、并行效率的影响,实验验证局域并行效率,实现对并行处理框架的优化,从而建立一种解决复杂地学问题的并行开发协作模式,为地理栅格数据的高效处理提供一种新的解决方法和技术支撑。本文的主要研究成果体现在如下几个方面： (1)提出了一种大规模地理栅格数据并行处理算法的并行解耦方法以MP工和MP作为基础并行环境,将地理栅格计算部分和并行计算支撑部分分别进行抽象和封装,作为相互协作的部件予以松耦合地装配和执行,有效地分离并行计算体系与地学问题的强耦合。 (2)构建了协作式大规模栅格数据的并行处理框架在地理栅格数据并行处理机制及并行解耦方法研究的基础上,建立了适合于协同开发的数据块分块、分发、缝合的数据类模型,为实现地学栅格数据协同并行奠定了基础；建立了核心算法的封装类模型及开发策略,实现了代码开发与算法细节分离,保证了并行计算与分析应用工作的协同。 (3)提出了基于并行解耦思想的地理栅格数据全局计算并行化方法针对内存一次无法装载的大型栅格地理数据却要进行地理栅格数据的全局计算,分析地学处理算法原理,基于CBGRDPPF框架利用对数据的横向、纵向划分、过程数据的分块暂存等策略,使各个并行进程在占用有限的内存空间情况下,分块依次处理整个栅格数据,大大降低了开发算法程序的复杂度,从而实现复杂并行地理数据计算任务高效并行。 (4)提出了基于并行解耦思想的地理栅格数据动态计算并行化方法动态计算主要是指一些栅格数据的聚类算法步骤未知,计算过程动态迭代。本文通过FCM算法为代表的聚类算法,基于CBGRDPPF框架,通过数据多策略的划分、序列化读取、计算同步和广播机制,实现一种动态计算的地理栅格数据并行处理方法,从而解决了不平衡计算量的并行化问题。
[Abstract]:The new generation of data acquisition technology, represented by high resolution remote sensing satellite, has made great progress. Geographic grid data has been improved in time and space resolution, data type, covering area and so on. It provides more data information for geographic applications. However, these improvements have also brought the geometric growth of data, but it also leads to traditional data. The remote sensing data processing method can not meet the needs of large-scale geographic grid data calculation and analysis. Therefore, it is of great significance to study the high performance computing methods and systems for large-scale geographic grid data, and to improve the efficiency of development and the ability to solve complex geological problems.
In order to solve the problem of large scale grid geographic data parallel processing, this paper systematically studies the parallel processing method of large-scale geographic grid data under high performance computing architecture, introduces MPI (Message Passing Interface) and MP (Multi processing) as the basic parallel environment, and studies the necessary functions of the grid data processing algorithm program. The necessary process, the generality of all algorithms, the idea of design pattern, a system framework that conforms to the basic principles of object-oriented programs, and a parallel processing framework for collaborative large-scale raster data (Cooperative Big Geographic Raster Data Parallel Processing Framework, CBGRDPPF), combined with geographic grid. With the characteristics of data parallel type and complexity, the cooperative processing method and technology of geoscience grid data processing task under this framework are discussed, and the influence of different parameters and environment on its running speed and parallel efficiency is analyzed. The experiment verifies the local parallel efficiency and the optimization of parallel processing framework. The parallel development and collaboration model for solving complex Geosciences problems provides a new solution and technical support for the efficient processing of geographic grid data. The main research results of this paper are reflected in the following aspects:
(1) a parallel decoupling method for large scale grid data parallel processing algorithm is proposed.
Using MP and MP as the basic parallel environment, the geographic grid computing part and the parallel computing support part are abstracted and encapsulated respectively. As a cooperative component, it is assembled and executed loosely coupled and effectively separates the strong coupling of the parallel computing system and the geoscience problem.
(2) build a parallel processing framework for collaborative large-scale raster data.
On the basis of the parallel processing mechanism of geographic grid data and the research of parallel decoupling method, a data class model which is suitable for collaborative development of data block, distribution and suture is established, which lays the foundation for the realization of geoscience grid data coordination and parallel. The package model and development strategy of the core algorithm are established, and the code development and calculation are realized. Separation of the method details ensures the synergy of parallel computation and analysis application.
(3) a parallel computing method for global raster data based on parallel decoupling is proposed.
In view of the large grid geographic data that can not be loaded once, the global calculation of geographic grid data should be carried out, the principle of the analysis of geoscience processing algorithm is analyzed. Based on the CBGRDPPF framework, the strategies of lateral, longitudinal division of the data and the partitioning of process data are used to make the parallel processes block under the limited memory space. Processing the whole raster data in a row, greatly reduces the complexity of the development algorithm program, and realizes efficient parallel computing of complex parallel geographic data.
(4) a parallel computing method for dynamic computation of geographic raster data based on parallel decoupling is proposed.
The dynamic calculation mainly refers to the unknown clustering algorithm of some grid data and the dynamic iteration of the calculation process. In this paper, the clustering algorithm, represented by the FCM algorithm, is based on the CBGRDPPF framework, through the division of multiple strategies, serialization reading and computing synchronization and broadcasting mechanism, and a dynamic computation of the parallel processing of geographic grid data is realized. The problem of parallelization of unbalanced computation is solved.
【学位授予单位】：首都师范大学
【学位级别】：博士
【学位授予年份】：2014
【分类号】：P208

【参考文献】