天文大数据挑战与实时处理技术

发布时间：2018-01-13 05:31

本文关键词：天文大数据挑战与实时处理技术　出处：《计算机研究与发展》2017年02期 　论文类型：期刊论文

【摘要】：超大型天文观测技术的出现不仅能够让研究人员观测到新的天文现象,更能用于验证已有物理模型的正确性.这些最新天文成果的发现是建立在海量天文数据的近乎实时产生、管理与分析的基础上,因此给目前的数据管理系统带来了新的挑战.以我国自主研发的地基广角相机阵(the ground-based wide-angle camera array,GWAC)天文望远镜为例,15s的采样和处理周期都处于短时标观测领域的世界前列,但却对数据管理系统提出了很多问题,包括多镜头并行输出数据管理、实时瞬变源发现、当前观测夜数据的秒级查询、数据持久化和快速离线查询等.基于上述问题,设计了分布式GWAC数据模拟生成器用于模拟真实GWAC数据产生场景,并基于产生的数据特性,提出一种两级缓存架构,使用本地内存解决多镜头并行输出、实时瞬变源发现,使用分布式共享内存实现秒级查询.为了平衡持久化和查询效率,设计一种星表簇结构将整个星表数据划分后聚集存储.根据天文需求特点,设计基于索引表的查询引擎能从缓存和星表簇以较小的代价对星表数据查询.通过实验验证,当前方案能够满足GWAC的需求.
[Abstract]:The emergence of super-large astronomical observation technology not only enables researchers to observe new astronomical phenomena. It can be used to verify the correctness of existing physical models. The discovery of these latest astronomical results is based on the near real-time generation, management and analysis of massive astronomical data. Therefore, it brings new challenges to the current data management system. The ground-based wide-angle camera array. The sampling and processing period of 15s for the GWAC telescope is in the forefront of the world in the field of short-time scale observation, but many problems have been raised for the data management system. It includes multi-shot parallel output data management, real-time transient source discovery, second level query of current observation night data, data persistence and fast off-line query. A distributed GWAC data simulation generator is designed to simulate the real GWAC data generation scene. Based on the characteristics of the generated data, a two-level cache architecture is proposed. Using local memory to solve multi-shot parallel output, real-time transient source discovery, distributed shared memory to achieve second-level query, in order to balance persistence and query efficiency. A cluster structure is designed to divide the data of the whole catalog and store them together, according to the characteristics of astronomical requirements. A query engine based on index table is designed to query the catalog data from cache and star cluster at a lower cost. The experimental results show that the current scheme can meet the requirements of GWAC.
【作者单位】：中国人民大学信息学院;清华大学计算机科学与技术系;中国科学院国家天文台;
【基金】：国家重点研发计划项目(2016YFB1000602)~~
【分类号】：P11;TP311.13
【正文快照】： 随着各种最新观测技术的出现,天文领域迎来了信息爆炸的时代,而该时代的第一波浪潮就是天文大数据的管理.进入21世纪,天文学已经进入了一个信息丰富的大数据时代,天文数据正在以TB量级甚至PB量级的速度快速增长.2000年斯隆数字巡天(Sloan digital sky survey,SDSS)项目启动时,

【相似文献】