基于NoSQL技术的MUSER数据归档与发布系统的设计与实现

发布时间：2018-12-18 05:25

【摘要】：随着高空间、高时间、高频率分辨率的观测设备不断投入使用,天文观测产生的数据飞速增长。传统的科学数据归档和处理方法远远不能满足海量天文数据的需求。为了提高现代天文观测的科学产出效率,必须探索新的数据归档与发布方法。本文以明安图超宽频谱射电日像仪(Mingantu Ultrawide SpEctral Radioheliograph, MUSER)项目为研究背景,随着MUSER项目进入观测阶段,如何快速有效地归档其多波段多通道观测产生的海量数据成为急需解决的问题。同时,为便于观测数据的共享,在归档的基础上应采取有效的方式对观测信息进行发布。本研究的工作主要包括四个方面：(1)通过基于NoSQL技术的FastBit技术和传统的关系型数据库MySQL技术在建立索引、查询、插入数据及归档空间大小上的对比实验,得出FastBit更适合MUSER的实际索引和检索需求。(2)利用模拟数据设计并实现了MUSER项目的数据归档系统原型,包括FITS、UV-FITS格式和PNG格式的数据归档。FITS文件在归档的同时建立索引,便于后期的数据检索。PNG格式的数据归档应用于项目的发布,基于MUSER按天观测的特性,归档的层级目录按天定时创建。(3)利用模拟数据实现了将MUSER原始数据处理生成的太阳图像与其相关观测信息的融合,生成的图像用于MUSER的发布系统原型,进行效果显示。(4)设计并实现了MUSER项目的数据发布系统原型,其中的发布包括每隔10min的图像发布和每日中天时间图像的动态发布。同时,发布系统实现了指定的时间间隔的动态效果展示。本文研究和实验的数据归档和发布系统原型,能应用在MUSER的实际归档与发布系统中,对其提供技术支撑。
[Abstract]:With the use of high space, high time and high frequency resolution observation equipment, astronomical observation data is increasing rapidly. Traditional scientific data archiving and processing methods can not meet the needs of massive astronomical data. In order to improve the efficiency of scientific output of modern astronomical observations, a new method of data archiving and publishing must be explored. In this paper, with the background of the (Mingantu Ultrawide SpEctral Radioheliograph, MUSER) project of the ultra-wide spectrum radio heliograph of the Ming Antu, with the entry of the MUSER project into the observation stage, How to quickly and effectively file the massive data generated by multi-band and multi-channel observations has become an urgent problem. At the same time, in order to facilitate the sharing of observation data, we should adopt an effective way to publish the observation information on the basis of archiving. The work of this study mainly includes four aspects: (1) the comparative experiment of establishing index, querying, inserting data and archiving space through FastBit technology based on NoSQL technology and MySQL technology of traditional relational database. It is concluded that FastBit is more suitable for the actual index and retrieval requirements of MUSER. (2) the prototype of MUSER project data archiving system, including FITS, is designed and implemented by using simulated data. Data archiving in UV-FITS format and PNG format. FITS files are indexed at the same time in order to facilitate the later data retrieval. PNG format data archiving is applied to the release of projects. Based on the characteristics of MUSER daily observation, The archived hierarchical catalogue is created on a daily basis. (3) using simulated data, the solar image generated by MUSER raw data processing is fused with its related observation information, and the generated image is used in the prototype of the MUSER publishing system. (4) the prototype of the MUSER project data publishing system is designed and implemented, which includes the release of every 10min image and the dynamic distribution of the daily mid-day image. At the same time, the release system realizes the dynamic effect display of the specified time interval. The prototype of data archiving and publishing system studied and experimented in this paper can be applied to the actual archiving and publishing system of MUSER and provide technical support to it.
【学位授予单位】：昆明理工大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：P111.41;TP311.13

【参考文献】