基于计算存储一体化策略的遥感数据高性能计算研究及应用

发布时间：2018-02-04 09:20

本文关键词： 高性能计算海量遥感数据集群计算计算存储一体化分布式存储　出处：《河南大学》2014年硕士论文　论文类型：学位论文

【摘要】：高性能计算机的迅速发展为海量遥感数据处理提供了强大的计算资源，基于集群及计算存储一体化的高性能计算已成为目前最主要的计算平台。在集群环境下设计并实现海量遥感数据并行处理系统是提高遥感数据处理速度的必然选择。随着高分辨率及传感器类型的增加，遥感数据TB级的增长对高性能遥感数据处理平台的I/O要求造成非常大的压力，包括服务器磁盘的读写压力，服务器网段的传输压力等，为了最大化发挥分布式并行优势，在保证系统安全稳定的前提下，本文采用了计算存储一体化策略，以减少并行处理系统中的数据传输。本文在分析了网格、云计算及集群计算等几种高性能计算的优缺点基础上，提出并实现了一种基于计算存储一体化的高性能集群架构，以实现对海量遥感数据的高效处理，并在卫星遥感基础共性产品一体化处理系统中得以应用。本文的工作主要体现在以下几点：（1）基于已有集群系统，深入研究遥感数据高性能计算处理系统，，针对海量遥感数据对处理平台I/O、服务器磁盘的读写及网络传输造成的压力等问题，采用把数据存储从原来集中到一个机器变成计算、存储一体化架构，最大限度降低系统架构的I/O传输，更充分利用了存储服务器的计算资源、计算服务器的存储资源，从而提高平台的整理处理时效。（2）采用了逻辑上静态、物理上可动态扩展的分布式存储模型来实现对海量遥感数据的存储及管理。引入虚拟磁盘空间（VDS）的概念来把切片分配到多个存储站点，并采用交叉备份方式确保系统的稳定性。（3）采用三级并行策略来最大化发挥高性能并行计算在大规模的遥感影像处理中发挥着的重要作用（4）实验验证。通过使用不同类型卫星的数据测试，从多个角度进行对比验证，证明本文提出的高性能集群架构并行处理能力和效率上均有提升。
[Abstract]:The rapid development of high-performance computers provides powerful computing resources for massive remote sensing data processing. The high performance computing based on the integration of cluster and storage has become the most important computing platform. It is an inevitable choice to design and implement the parallel processing system of massive remote sensing data in cluster environment to improve the speed of remote sensing data processing. With the increase of high resolution and sensor type, the increase of TB level of remote sensing data causes great pressure on the I / O requirement of high performance remote sensing data processing platform, including the pressure of reading and writing of server disk, the pressure of transmission of server network segment, etc. In order to maximize the advantage of distributed parallelism, under the premise of ensuring the security and stability of the system, this paper adopts the strategy of integration of computing and storage to reduce the data transmission in parallel processing system. On the basis of analyzing the advantages and disadvantages of grid, cloud computing and cluster computing, this paper proposes and implements a kind of high performance cluster architecture based on computing and storage integration, which can efficiently process the massive remote sensing data. And it is applied in the integrated processing system of satellite remote sensing basic commonality product. The work of this paper is mainly reflected in the following points:. (1) based on the existing cluster system, the high performance computing and processing system of remote sensing data is studied in depth. Aiming at the problems caused by the massive remote sensing data on the processing platform I / O, the read, write and transmission of server disk, and so on, By changing the data storage from the original centralized to a single machine into a computing, storage integration framework, the I / O transmission of the system architecture is minimized, and the computing resources of the storage server and the storage resources of the computing server are more fully utilized. In order to improve the platform finishing and processing time. The distributed storage model, which is logically static and dynamically scalable in physics, is used to store and manage the massive remote sensing data. The concept of virtual disk space (VDS) is introduced to distribute slices to multiple storage sites. Cross-backup is used to ensure the stability of the system. Using the three-level parallel strategy to maximize the high performance parallel computing plays an important role in large-scale remote sensing image processing. By using the data of different types of satellites, the parallel processing capability and efficiency of the proposed high performance cluster architecture are proved to be improved.
【学位授予单位】：河南大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP79

【参考文献】