云计算平台下遥感图像分类算法的并行化技术研究

发布时间：2018-01-13 13:25

本文关键词：云计算平台下遥感图像分类算法的并行化技术研究　出处：《南京理工大学》2017年硕士论文　论文类型：学位论文

【摘要】：高光谱遥感图像中包含着丰富的空间信息和光谱信息,使其在地物识别和分类方面较其他遥感图像更具优势,因此它在军事侦查、资源勘探和环境监测等领域广泛应用。高光谱遥感图像分类是高光谱图像处理的重要内容之一,由于高光谱遥感图像具有高维、波段多、数据量大的特点,现有串行分类算法计算复杂度较高,算法的实时性并不好,同时随着遥感器分辨率的不断提升,高光谱遥感图像数据量呈几何式增长,现有的单机计算平台难以处理产生的海量高光谱遥感图像数据。云计算技术具备分布式存储和分布式计算的特点,能够有效解决高光谱遥感图像分类的单机瓶颈问题。云计算平台下任务调度算法对任务的执行性能有着非常重要的影响,合适的调度策略可以提高任务的执行速度。因此本文对云计算平台下基于空间相关性正则化稀疏表示的高光谱遥感图像分类方法SCSRC的并行化和任务调度展开了深入研究,主要工作有:(1)常规遥感图像的传统分类算法并不能满足大数据量的高光谱遥感图像的分类需求,因此有学者提出了基于空间相关性正则化的稀疏表示分类方法SCSRC,这种分类方法不仅利用了高光谱遥感图像的光谱信息,而且添加了图像空间相邻数据之间的信息,取得了较好的分类效果。本文首先研究并在单机上实现了 SCSRC方法,并通过实验分析了该方法的时间性能,为后续云计算平台下的并行化研究提供对比依据。(2)由于SCSRC算法的计算复杂度较高,并且还受到单机尺度的限制,因此在分析SCSRC算法性能瓶颈的基础上,分别在Hadoop和Spark平台下设计了 SCSRC的并行化方法MR_SCSRC和SK_SCSRC。在MR_SCSRC算法中首先设计了基于外积法实现矩阵相乘的MapReduce方法,然后从降低Map阶段IO次数、合并计算逻辑和实现本地化规约三个方面对算法进行了优化。鉴于在Hadoop的基础上发展而来的Spark更适用于迭代计算,参考MR_SCSRC的实现思路,进一步设计了基于Spark RDD编程模型的SK__SCSRC方法。最后通过实验给出了 SK_SCSRC算法的加速比和扩展比,并将MR_SCSRC和SK_SCSRC进行了对比。(3)Hadoop现有的任务调度算法只是考虑了数据本地性,忽视了集群的负载均衡性,为此基于遗传算法设计实现了兼顾作业完成时间和集群负载均衡性的任务调度器StaticGAtaskScheduler,详细给出了该调度器在Hadoop平台下的加载调度验证,最后通过调度MR_SCSRC算法验证此调度器下算法的性能提升。
[Abstract]:Hyperspectral remote sensing image contains spatial information and spectral information, compared to the other remote sensing image has more advantages in object recognition and classification, so it is widely used in military reconnaissance, resource exploration and environmental monitoring. The hyperspectral remote sensing image classification is one of the important contents of the hyperspectral image processing, due to the high remote sensing image with high dimension, multi band, large amount of data, the existing serial classification algorithm has high computational complexity, real-time algorithm is not good, at the same time as the remote sensor resolution upgrade, hyperspectral remote sensing image data amount of exponential growth, the existing single amount of hyperspectral remote sensing image data to the platform processing. Cloud computing technology has the characteristics of distributed storage and distributed computing, which can effectively solve the hyperspectral remote sensing image classification of single cloud bottlenecks. Is the performance of task scheduling algorithm under the platform of the task is very important, appropriate scheduling strategy can improve the speed of execution tasks. Parallel task scheduling and the calculation of SCSRC classification method of hyperspectral remote sensing image spatial correlation regularization based on the sparse representation of cloud platform are studied, the main work a: (1) conventional remote sensing image classification and traditional classification algorithm can not meet the demand of hyperspectral remote sensing image data, so some scholars proposed a classification method of SCSRC sparse spatial correlation based on regularization, this classification method not only uses the spectral information of hyperspectral remote sensing image, and add a space between the image the data of the adjacent information, to obtain a better classification. This paper studies and implements the SCSRC method on a single machine, and through experimental analysis The time performance of the method, provides the basis for subsequent research on parallel cloud computing platform. (2) due to the computational complexity of the SCSRC algorithm is higher, and also by the single scale restricted, so based on the analysis of the SCSRC algorithm performance bottleneck, respectively in Hadoop and Spark platform to design the parallelization of SCSRC the methods of MR_SCSRC and SK_SCSRC. in MR_SCSRC algorithm is first designed MapReduce method of outer product implementation of matrix multiplication based on Map, then from the lower stage of IO times, and realize the localization of protocol logic three aspects to optimize the algorithm combined. In view of the development and on the basis of Hadoop Spark is more suitable for the iterative calculation, ideas the reference MR_SCSRC, further design method of Spark SK__SCSRC based on RDD programming model. Finally, the experiment shows the SK_SCSRC algorithm ratio and expansion ratio, and MR_S Compared with CSRC and SK_SCSRC. (3) Hadoop existing task scheduling algorithms only consider the data locality, ignoring the load balance of cluster, the genetic algorithm is designed to achieve balanced work load balancing of the StaticGAtaskScheduler task scheduler and cluster based on time, with the load scheduling under the Hadoop platform to verify the scheduler this, finally through the scheduling algorithm MR_SCSRC verify the performance of the scheduler algorithm upgrade.

【学位授予单位】：南京理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP751

【相似文献】