基于Hadoop的云计算模型研究及气象应用

发布时间：2018-01-19 06:24

本文关键词： 云计算 Hadoop MapReduce GIS 气象信息分析与处理　出处：《南京信息工程大学》2012年硕士论文　论文类型：学位论文

【摘要】：互联网技术的飞速发展,带来了数据量的井喷式增长,传统技术在处理这些海量数据遇到了不可避免的效率瓶颈。尤其在气象、遥感、地质灾害监测等特殊行业,能否及时、安全、高效地处理其中的海量数据,关系到人民的财产安全和社会和谐稳定。云计算应运而生,其处理海量数据方面有着技术固有的优势,其中Apache Hadoop开源云计算平台正受到知名互联网公司和数据库厂商的支持；Hadoop技术正越来越受到国内外研究者们的关注,也成为海量数据处理技术研究热点。本文旨在通过对Hadoop中的HDFS和MapReduce进行深入研究和探索,并结合气象灾害监测及评估系统工程开发,将Hadoop应用于基于GIS的气象实际业务系统中,并进行分析研究。本文的主要工作如下：首先对气象数据的特点进行分析,并指出将气象数据直接存储于Hadoop中存在的问题。根据气象数据的特点将设计基于Trie树变形的文件合并算法。在此基础上通过实验验证,数据高效率处理、安全等方面得到有效的提升和保障。其次,设计并实现基于Hadoop的海量气象异构数据存储和计算框架,针对气象数据中实时提取有效数据的性能效率、数据保密安全、灾害备份等问题,利用HDFS存储气象数据以及利用MapReduce构建并行算法,将提出基于Hadoop的海量气象异构数据存储和处理框架。并通过实际业务中运行,将验证在并发数据处理上的效果,达到能有效降低成本和保证数据的安全的目标。本文通过研究的模型与实际业务应用相结合,利用前沿的分布式技术来很好的满足项目中的需要,并将模型部署到实例当中,用实际运行结果来检验模型的实用价值。这些工作对大数据处理的研究有着重要的理论价值和实用价值。
[Abstract]:With the rapid development of Internet technology, the amount of data blowout growth, traditional technology in processing these massive data encountered an inevitable efficiency bottleneck, especially in meteorology, remote sensing. Geological hazard monitoring and other special industries, whether timely, safe, efficient processing of the massive data, related to people's property security and social harmony and stability. Cloud computing emerged as the times require. The Apache Hadoop open source cloud computing platform is supported by well-known Internet companies and database vendors. Hadoop technology has been paid more and more attention by researchers at home and abroad, and has also become a hot spot of mass data processing technology. The purpose of this paper is to study and explore the HDFS and MapReduce in Hadoop, and to develop the meteorological disaster monitoring and evaluation system. The application of Hadoop in the meteorological practical operational system based on GIS is analyzed and studied. The main work of this paper is as follows: Firstly, the characteristics of meteorological data are analyzed. The problem of storing meteorological data in Hadoop directly is pointed out. According to the characteristics of meteorological data, a file merging algorithm based on Trie tree deformation is designed. Data efficient processing, security and other aspects are effectively promoted and guaranteed. Secondly, the storage and calculation framework of massive meteorological heterogeneous data based on Hadoop is designed and implemented. Aiming at the performance efficiency of extracting effective data in real-time, data security, disaster backup and so on. HDFS is used to store meteorological data and MapReduce is used to construct parallel algorithm. The storage and processing framework of massive meteorological heterogeneous data based on Hadoop will be put forward, and the effect of concurrent data processing will be verified by running in the actual business. To achieve the goal of effectively reducing costs and ensuring data security. In this paper, the model is combined with the actual business application, using the frontier distributed technology to meet the needs of the project, and the model is deployed to the example. The practical value of the model is verified by the practical operation results, which are of great theoretical and practical value to the study of big data's treatment.
【学位授予单位】：南京信息工程大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：P413;TP3

【相似文献】