Argo剖面浮标数据异常检测方法研究

发布时间：2019-04-02 01:14

【摘要】：Argo剖面浮标是目前唯一实时获取全球上层海洋三维观测数据的手段,其观测的数据反映了海洋立体温盐分布的情况,对研究海洋环流、全球气候变化、海洋分析预报系统提供了数据基础,具有极其重要的应用意义与科学价值。本文以Argo剖面浮标数据作为海洋大数据信息安全研究的切入点,针对Argo剖面浮标数据受环境、设备等不确定因素的影响导致数据出现异常的问题,结合Argo剖面浮标数据具有数据量大、地域性、非线性分布、离散等特点,开展了Argo剖面浮标数据异常检测方法的深入研究,为提高Argo剖面浮标数据准确性、可靠性提供理论依据和技术手段。本文针对异常检测的训练阶段与异常检测阶段分别展开分析与研究。首先,在训练阶段,针对Argo剖面浮标数据文件格式复杂、数据量庞大的问题,提出基于MapReduce技术的Argo主剖面生成的AMPC(information fusion algorithm for Argo profile base on MapReduce and Principal Curves)算法。该算法采用MapReduce技术有效地提高执行效率,另外,将剖面信息以经纬度进行分类,增强剖面间关联性,凸显剖面地域特点,并在K-主曲线理论的基础上,使用不断添加拟合剖面点的方式生成主剖面,减少异常检测阶段的数据存储量,为剖面可能存在的点异常、上下文异常、集合异常提供参考依据。其次,在异常检测阶段,汲取基于“三倍标准差”准则异常检测方法与基于预测模型异常检测方法的优点,改进设计一种结合分段“三倍标准差”准则和k-近邻子剖面与主曲线预测方法的基于自适应异常阈值的异常检测方法。该方法以训练阶段生成的主剖面作为参考,综合当前待测剖面点对主剖面的偏离情况与剖面随深度变化的趋势给异常检测带来的影响,动态计算各剖面点的异常阈值,进一步提高异常检测方法的性能与检测效果。验证试验表明:通过全球Argo剖面浮标数据的验证,本文研究的针对Argo剖面浮标数据特点的异常检测方法高效地结合历史剖面数据,有效地规避静态阈值检测的片面性,具有良好的异常检测效果,异常检测的准确度得到明显提高。
[Abstract]:Argo profile buoy is the only real-time method to obtain three-dimensional observational data from global upper ocean. The observed data reflect the distribution of ocean temperature and salt in three-dimensional ocean. It is very important to study ocean circulation and global climate change. The ocean analysis and forecast system provides the basis of data, and has very important application significance and scientific value. This paper takes the Argo profile buoy data as the breakthrough point of the marine big data information security research, aiming at the problem that the Argo profile buoy data is influenced by the uncertain factors such as environment, equipment and so on, which leads to the abnormal data. According to the characteristics of Argo profile buoy data, such as large amount of data, region, non-linear distribution, discrete and so on, an in-depth study on anomaly detection method of Argo profile buoy data is carried out in order to improve the accuracy of Argo profile buoy data. Reliability provides theoretical basis and technical means. In this paper, the training phase and anomaly detection phase of anomaly detection are analyzed and studied respectively. Firstly, in the training stage, aiming at the complicated format and huge data volume of the Argo profile buoy data file, a AMPC (information fusion algorithm for Argo profile base on MapReduce and Principal Curves) algorithm based on MapReduce technology for the generation of Argo main profile is proposed. In addition, the section information is classified by latitude and longitude to enhance the correlation between sections and highlight the regional characteristics of the section, and on the basis of K-principal curve theory, the algorithm uses MapReduce technology to improve the efficiency of execution effectively. The main section is generated by continuously adding fitting section points to reduce the amount of data stored in the anomaly detection phase, which provides reference for possible point exceptions, context exceptions, and set exceptions in the profile. Secondly, in the anomaly detection phase, the advantages of the anomaly detection method based on the "triple standard deviation" criterion and the anomaly detection method based on predictive model are drawn. An improved anomaly detection method based on adaptive anomaly threshold is designed, which combines the segmented "triple standard deviation" criterion and the prediction method of k-nearest neighbor profile and principal curve. In this method, the main section generated during the training stage is taken as a reference, and the deviation of the current section point to the main section and the influence of the variation trend of the profile with the depth on the anomaly detection are integrated, and the abnormal threshold values of each section point are calculated dynamically. Further improve the performance and detection effect of the anomaly detection method. The verification test shows that: through the verification of global Argo profile buoy data, the anomaly detection method based on the characteristics of Argo profile buoy data studied in this paper effectively combines historical profile data, and effectively avoids the one-sidedness of static threshold detection. It has good effect of anomaly detection, and the accuracy of anomaly detection is improved obviously.
【学位授予单位】：桂林电子科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：P715.2;TP311.13

【相似文献】