海量极区空间物理数据组织与服务关键技术

发布时间：2018-05-12 03:45

本文选题：极区空间物理 + 科学数据管理　；参考：《华东师范大学》2017年硕士论文

【摘要】：极区空间物理主要研究南北两极的大气层、电离层和磁层中的物理现象,随着数字化科学仪器的发展,极区空间物理研究已成为数据密集型的科学研究。目前使用的观测仪器大致可以分为电离层、地磁和极光观测三类。不同观测仪器采集的数据的类型、结构各不相同,是典型的多维结构的科学数据,数据规模已经累积达到了 TB甚至PB级别。如何对这些海量的多维科学数据进行有效组织,并提供快速查询等数据服务以支持极区空间物理的研究,成为亟待解决的问题。针对关系数据库和NoSQL数据库的数据模型在表达多维科学数据上存在的先天缺陷和管理科学数据的高昂代价,本文基于Array数据模型设计并实现了海量极区空间物理数据的组织与服务框架。Array数据模型专为多维结构的数据设计,可以容易地表达多维科学数据,对多维科学数据的运算支持良好。本文对Array数据模型进行了扩展,增加了对元数据的支持,设计了 ArrayPlus数据模型用以表达极区空间物理数据,通过元数据管理器的设计,基于Array数据库Rasdaman提供了对ArrayPlus数据模型的支持。考虑到极光图像数据载入数据库的代价过高,本文设计了 Aurora Vault来实现极光图像的原位处理功能。同时,针对Rasdaman在数据分发功能上的不足,本文设计了分布式调度器,支持对Aurora Vault的分布式调度。在此基础上,面向极区空间物理数据的管理需求,设计了数据服务,提供了数据导入、可视化、基本统计、下载、极光图像处理等功能,使用Java语言实现了海量极区空间物理数据的组织与服务原型系统。最后,本文比较分析了基于关系数据库、基于NoSQL数据库和基于本文扩展的Rasdaman的数据访问代价,通过地磁、极光、宇宙噪声三类极区空间物理数据的8个典型查询案例,选取MySQL、Mongodb、HBase与本文的研究进行了比较实验。实验过程表明,基于本文扩展的Rasdaman实现查询需求的代价明显小于其余三者;实验结果表明,基于本文扩展的Rasdaman的查询性能明显优于其余三者,也进一步说明了本文设计的海量极区空间物理数据的组织与服务框架的有效性。
[Abstract]:Polar space physics mainly studies the physical phenomena in the atmosphere, ionosphere and magnetosphere of the northern and southern poles. With the development of digital scientific instruments, polar space physics research has become data-intensive scientific research. The observational instruments used at present can be divided into ionospheric, geomagnetic and auroral observations. The types and structures of data collected by different observation instruments are typical multi-dimensional scientific data. The scale of data has accumulated to TB or PB level. How to organize these massive multi-dimensional scientific data effectively and provide fast query and other data services to support the research of polar spatial physics has become an urgent problem. Aiming at the inherent defects in the expression of multidimensional scientific data and the high cost of managing scientific data, the data models of relational database and NoSQL database are used to express the multi-dimensional scientific data. Based on the Array data model, this paper designs and implements the organization and service framework of massive polar spatial physical data. The data model is specially designed for multidimensional data structure, which can easily express multidimensional scientific data. Good operation support for multidimensional scientific data. In this paper, the Array data model is extended to support metadata, and the ArrayPlus data model is designed to express polar spatial physical data. Rasdaman provides support for ArrayPlus data model based on Array database. Considering the high cost of loading auroral image data into database, Aurora Vault is designed to realize in situ processing of aurora image. At the same time, aiming at the deficiency of Rasdaman in data distribution, this paper designs a distributed scheduler to support the distributed scheduling of Aurora Vault. On this basis, to meet the management requirements of polar spatial physical data, a data service is designed to provide functions such as data import, visualization, basic statistics, download, aurora image processing, etc. The prototype system of organizing and serving massive polar spatial physical data is implemented by using Java language. Finally, this paper compares and analyzes the data access cost based on relational database, NoSQL database and extended Rasdaman. Eight typical query cases of spatial physical data in polar region of geomagnetic, auroral and cosmic noise are analyzed. A comparative experiment was carried out between MySQL MongodbHase and this study. The experimental results show that the cost of implementing query requirements based on the extended Rasdaman is obviously lower than that of the other three, and the experimental results show that the query performance of the extended Rasdaman is obviously better than that of the other three. Furthermore, the validity of the organization and service framework of the massive polar spatial physical data designed in this paper is further illustrated.
【学位授予单位】：华东师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【参考文献】