基于Hadoop平台的气象数据挖掘研究

发布时间：2019-05-18 18:56

【摘要】：随着互联网、计算和存储技术的快速发展,气象部门存储的气象数据量日益剧增,我国每年新增的气象资料达到PB量级,同时气象数据类型相对复杂,这使得传统的数据存储和处理技术不能很好解决目前用户的需求。当前国内外许多研究团队都致力于归纳分析这些海量数据,并从中挖掘出具有现实意义的气象规律或模式。近几年,云计算技术作为互联网领域的新产物,它为海量数据存储和处理提供了新的契机,它在海量数据挖掘技术领域中具有显著的优势,且已经得到了广泛的应用。云计算的总体思路是利用网络互联技术将若干台计算机连接在一起,实现对资源集中管理和统一调度,相当于形成一个资源池,以此按用户需求为其提供服务。Hadoop技术作为云计算软件平台中处理数据的一种解决方案,它具有高容错、高吞吐量、低成本等特点,将传统的数据挖掘技术移植到Hadoop云平台上,实现了低成本高效率的数据挖掘过程,这也已经成为气象数据挖掘研究方向的一个趋势。本文深入研究了基于Hadoop平台的数据挖掘方法以及气象资料的特性,针对现有的贝叶斯分类数据挖掘方法存在的一些不足,结合Hadoop云平台处理海量数据的优势,提出了基于MapReduce的贝叶斯网络分类改进算法,为此,本文主要做了如下研究:(1)针对气象数据的大规模特性,本文采用了 Hadoop平台对气象原始数据集进行预处理和计算任意两个特征属性之间的相关系数,利用相关分析技术选取预测属性,在一定程度上减少模型训练的复杂度。(2)分析典型气象数据挖掘分类算法的优劣性,针对气象数据的关联特性,本文采用了贝叶斯网络分类算法,它的提出就是为了解决事物的不确定性和关联性,较其他分类算法更适合气象数据分析。(3)在贝叶斯分类模型训练过程中,采用了精度评估,对不满足精度要求的模型采用迭代训练,并不断修改模型参数,以便获得较优的分类模型,并对测试集进行分类实验。实验结果表明,改进算法较现有的算法在计算效率和性能上均有一定的提高。
[Abstract]:With the rapid development of Internet, computing and storage technology, the meteorological data stored by meteorological departments is increasing day by day. The new meteorological data in China reach the order of PB every year, and the types of meteorological data are relatively complex. This makes the traditional data storage and processing technology can not solve the needs of current users. At present, many research teams at home and abroad are committed to summing up and analyzing these massive data, and mining out the meteorological laws or patterns of practical significance. In recent years, cloud computing technology, as a new product in the field of Internet, provides a new opportunity for mass data storage and processing. Cloud computing technology has significant advantages in the field of massive data mining technology, and has been widely used. The overall idea of cloud computing is to use network interconnection technology to connect several computers together to achieve centralized management and unified scheduling of resources, which is equivalent to the formation of a resource pool. Hadoop technology, as a solution to deal with data in cloud computing software platform, has the characteristics of high fault tolerance, high throughput, low cost and so on. Hadoop technology transports traditional data mining technology to Hadoop cloud platform. The process of low cost and high efficiency data mining has been realized, which has become a trend in the research direction of meteorological data mining. In this paper, the data mining method based on Hadoop platform and the characteristics of meteorological data are deeply studied. In view of some shortcomings of the existing Hadoop classification data mining methods, combined with the advantages of Hadoop cloud platform in dealing with massive data, An improved classification algorithm based on MapReduce is proposed. For this reason, this paper mainly does the following research: (1) aiming at the large-scale characteristics of meteorological data, In this paper, Hadoop platform is used to preprocess the meteorological original data set and calculate the correlation coefficient between any two feature attributes, and the prediction attributes are selected by using correlation analysis technology. To a certain extent, the complexity of model training is reduced. (2) the advantages and disadvantages of typical meteorological data mining classification algorithms are analyzed. According to the association characteristics of meteorological data, this paper adopts the Bayesian network classification algorithm. In order to solve the uncertainty and relevance of things, it is more suitable for meteorological data analysis than other classification algorithms. (3) in the training process of Bayesian classification model, the accuracy evaluation is adopted. Iterative training is used for the model that does not meet the accuracy requirements, and the model parameters are constantly modified in order to obtain a better classification model, and the classification experiment of the test set is carried out. The experimental results show that the improved algorithm has a certain improvement in computational efficiency and performance compared with the existing algorithms.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】