基于HBase的高能物理数据存储及分析平台

发布时间：2018-07-03 20:39

本文选题：高能物理数据 + 大数据　；参考：《计算机工程》2015年06期

【摘要】：高能物理对撞机产生数百亿计的物理事例,而物理分析则是从中选取几千个有意义的事例,该分析过程是一个典型的大数据处理及数据挖掘应用。由此,设计高效的数据结构、存储及访问机制,快速挑选出有意义的物理事例十分重要。介绍事例的数据结构、存储和处理技术,分析高能物理数据的特点,提出一种以HBase,ROOT,BEAN及Map Reduce为基础的新型高能物理数据存储及处理技术系统。利用HBase存储数据、Map Reduce实现并行处理,选择ROOT和BEAN作为高能物理分析框架,并给出具体设计与实现方案。测试结果表明,与传统高能物理数据存储系统相比,该系统具有更快的数据处理速度,当预筛选服务生效时能够更加有效地利用I/O和CPU资源。
[Abstract]:The high-energy physics collider produces tens of billions of physical examples, and the physical analysis is to select thousands of meaningful examples. The analysis process is a typical application of big data processing and data mining. Therefore, it is very important to design efficient data structures, storage and access mechanisms, and quickly select meaningful physical examples. This paper introduces the data structure, storage and processing technology of examples, analyzes the characteristics of high energy physics data, and puts forward a new high energy physics data storage and processing system based on HBaseROOTBEAN and Map reduce. The parallel processing is realized by using HBase storage data mapReduce. Root and beam are selected as the analysis framework of high energy physics, and the concrete design and implementation scheme are given. The test results show that compared with the traditional high-energy physical data storage system, the system has faster data processing speed and can utilize the I / O and CPU resources more effectively when the pre-filter service becomes effective.
【作者单位】：中国科学院高能物理研究所;中国科学院大学;
【基金】：国家自然科学基金资助项目(11375223,11375221);国家自然科学基金A3前瞻计划基金资助项目(61161140454) 国家自然科学基金委员会-中国科学院大科学装置联合基金资助项目(11179020)
【分类号】：TP333

【共引文献】