Spark下遥感大数据特征提取的加速策略

发布时间：2018-04-09 10:01

本文选题：Spark分布式内存计算框架　切入点：Hadoop分布式文件系统　出处：《计算机工程与设计》2017年12期

【摘要】：提出一种基于Spark分布式内存计算框架的遥感大数据特征提取策略。采用Landsat8为数据源,以计算归一化植被指数(NDVI)、差值植被指数(DVI)、比值植被指数(RVI)为例开展实验。实验结果表明,在相同硬件环境、处理任务、数据量的条件下,Spark处理遥感大数据的速度较单机模式下的处理遥感大数据提升了约2倍,基于Hadoop分布式文件系统(HDFS)处理模式较Spark-standalone处理模式处理速度提升了约1.2倍,基于Spark下的HDFS存储模式下,栅格切分遥感大数据较非栅格切分处理速度提高了约1.5倍。
[Abstract]:A remote sensing big data feature extraction strategy based on Spark distributed memory computing framework is proposed.Using Landsat8 as the data source, the experiments were carried out with the examples of the calculation of normalized vegetation index (NDVI), the difference vegetation index (DVI) and the ratio vegetation index (RVI).The experimental results show that the speed of Spark processing remote sensing big data under the same hardware environment, processing task and data volume is about 2 times faster than that of single machine mode.The processing speed of distributed file system (Hadoop) based on Hadoop is about 1.2 times faster than that of Spark-standalone. In HDFS storage mode based on Spark, the processing speed of remote sensing of grid segmentation is about 1.5 times faster than that of non-grid segmentation.
【作者单位】：新疆大学软件学院;
【基金】：国家自然科学基金项目(61562086;61462079;61363083;61262088) 新疆“万人计划”后备基金项目(wr2015bj01) 新疆自治区研究生科研创新基金项目(XJGRI2016029)
【分类号】：TP751

【相似文献】