MapReduce模型下的图像并行化处理研究

发布时间：2018-02-09 08:55

本文关键词： Hadoop MapReduce模型图像并行化　出处：《西安科技大学》2017年硕士论文　论文类型：学位论文

【摘要】：近年来,伴随着云计算和大数据的兴起,网络中各应用领域所产生的数据量快速增长,已经达到PB级别甚至更高。在这些数据中,图像大数据的处理与存储已经成为各界研究的热点。MapReduce技术是一种高可靠的并行编程框架,常用于进行大数据量的并行计算,对复杂的集群环境问题有着很好的解决方案。Hadoop平台的核心之一MapReduce模型就是利用该项技术处理海量数据,目前已取得了较好成效,但不足的是对于图像文件的处理,尤其是海量小图像文件的研究还不成熟。本文针对于此问题,探讨了在MapReduce模型下图像的并行化处理,给出了一种可以作为处理海量小图像文件的新的数据平台基础架构,主要研究的内容及研究成果如下:首先,本文综述了在大数据背景下,海量图像数据处理的研究背景、研究现状及意义,并介绍了 Hadoop生态系统,包括其核心HDFS系统和MapReduce框架技术。其次,详细设计了如何改进Hadoop系统,提出组合分片方法,提高了海量小图片的处理效率,扩展了 MapReduce软件框架,使之能够很好的支持并处理图像文件。最后,本文设计并实现了在MapReduce模型下,图像的并行K-means聚类算法分析、图像的并行Sobel边缘检测算法、扩展了图像在MapReduce模型下的并行化直方图提取等操作。通过实验验证,扩展后的MapReduce模型的可行性及处理图像文件的高效性。通过指标性能分析,验证了 MapReduce模型下进行图像并行化处理的有效性,为在Hadoop平台下处理海量大数据图像文件的应用提供了一种可行的解决方案。
[Abstract]:In recent years, with the rise of cloud computing and big data, the amount of data generated in various application areas of the network has grown rapidly, reaching the PB level or higher. The processing and storage of image big data has become a hot topic of research. MapReduce technology is a high reliable parallel programming framework, which is often used for parallel computing of large amount of data. There is a good solution to the complex cluster environment problem. One of the core of the Hadoop platform MapReduce model is to use this technology to process massive data. At present, it has achieved good results, but the lack of the image file processing, Especially, the research of massive small image files is not mature. In this paper, we discuss the parallelization of images under MapReduce model, and give a new data platform infrastructure which can be used to deal with large amount of small image files. The main research contents and results are as follows: firstly, this paper summarizes the research background, research status and significance of massive image data processing under the background of big data, and introduces the Hadoop ecosystem. It includes its core HDFS system and MapReduce framework technology. Secondly, how to improve the Hadoop system is designed in detail, and the combined slicing method is put forward, which improves the processing efficiency of massive small images and extends the MapReduce software framework. Finally, this paper designs and implements the parallel K-means clustering algorithm under the MapReduce model, and the parallel Sobel edge detection algorithm of the image. The parallel histogram extraction of image under MapReduce model is extended. The feasibility of the extended MapReduce model and the efficiency of processing image files are verified by experiments. The validity of parallel image processing based on MapReduce model is verified, which provides a feasible solution for processing massive big data image files on Hadoop platform.
【学位授予单位】：西安科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.41

【参考文献】