高能物理大数据挑战与海量事例特征索引技术研究

发布时间：2018-04-09 20:08

本文选题：高能物理　切入点：数据管理　出处：《计算机研究与发展》2017年02期

【摘要】：新一代高能物理实验装置的建成与运行,产生了PB乃至EB量级的数据,这对数据采集、存储、传输与共享、分析与处理等数据管理技术提出了巨大挑战.事例是高能物理实验的基本数据单元,一次大型实验即可产生万亿级的事例.传统高能物理数据处理以ROOT文件为基本存储和处理单位,每个ROOT文件可以包含数千至数亿个事例.这种基于文件的处理方式虽然降低了高能物理数据管理系统的开发难度,但物理分析仅对极少量的稀有事例感兴趣,这导致了数据传输量大、I?O瓶颈以及数据处理效率低等问题.提出一种面向事例的高能物理数据管理方法,重点研究海量事例特征高效索引技术.在这种方法中,将物理学家感兴趣的事例的特征量抽取出来建立专门的索引,存储在NoSQL数据库中.为便于物理分析处理,事例的原始数据仍然存放在ROOT文件中.最后,通过系统验证和分析表明,基于事例特征索引进行事例筛选是可行的,优化后的HBase系统可以满足事例索引的需求.
[Abstract]:The establishment and operation of a new generation of high-energy physics experimental devices have produced PB and EB order of magnitude data, which poses a great challenge to data acquisition, storage, transmission and sharing, analysis and processing and other data management technology.Example is the basic data unit of high energy physics experiment. A large experiment can produce trillion-scale examples.Traditional high energy physical data processing takes ROOT file as the basic storage and processing unit. Each ROOT file can contain thousands to hundreds of millions of examples.Although this kind of file-based processing method reduces the difficulty of developing high energy physics data management system, physical analysis is only interested in a few rare cases, which leads to the bottleneck of large amount of data transmission and the low efficiency of data processing.A case oriented high energy physics data management method is proposed.In this method, the features of the case of interest to physicists are extracted to build a special index and stored in the NoSQL database.For ease of physical analysis, the raw data of the case is still stored in the ROOT file.Finally, the system verification and analysis show that case filtering based on case feature index is feasible, and the optimized HBase system can meet the needs of case index.
【作者单位】：中国科学院高能物理研究所;西安交通大学计算机科学与技术系;中国科学院计算技术研究所;中国科学院信息工程研究所;
【基金】：国家重点研发计划项目(2016YFB1000604)~~
【分类号】：O572;TP315

【相似文献】