PetroV分布式数据存储与分析框架设计

发布时间：2018-05-16 09:51

本文选题：分布式存储与计算 + 八叉树　；参考：《石油地球物理勘探》2017年04期

【摘要】：随着三维地震数据采集、油气田开发等系列新技术带来的快速数据更新,大数据背景下三维数据体的高效存储与处理分析变得越来越复杂。在参考GFS(Google File System,谷歌文件系统)设计理念和消化其关键分布式处理技术的基础上,本文设计了PetroV(Petroleum Valuation,勘探部署决策一体化软件系统)最新软件架构并研发了系列大数据存储与分析的关键软件技术。利用三维空间下八叉树结构与编码的快速空间定位和多分辨率机制,实现对三维大数据体的结构分层、分块存储与二级内存缓存框架,支持并发访问和不同分辨率流式显示。由数据存取客户端、元数据服务和子数据块存取服务组成的八叉树分布式存储框架,屏蔽了基于八叉树切分后子体数据块在上百台计算机的后台冗余或备份存储事实,实现了几乎与单机文件系统接口一致的分布式文件存取接口;由任务执行客户端、任务管理服务和面向地震、数字岩心或测井数据的系列专业解释算法服务组成的分布式大数据分析框架,利用八叉树分块存储的特点实现"分而治之"并行编程模型,显著降低并行编程模型实现的复杂度。大数据下PetroV软件体系架构设计及衍生的系列专业软件版本,最终目的是希望能够推动应用导向与新技术发展深度结合,持续追求卓越,增强自主创新能力。
[Abstract]:With the rapid data updating brought by a series of new technologies such as 3D seismic data acquisition, oil and gas field development, the efficient storage and processing of 3D data under the background of big data has become more and more complicated. Based on the GFS(Google File system (Google File system) design concept and the digestion of its key distributed processing technologies, This paper designs the latest software architecture of PetroV(Petroleum validation (integrated software system for exploration and deployment decision) and develops a series of key software technologies for big data storage and analysis. Using the fast spatial location and multi-resolution mechanism of octree structure and coding in three-dimensional space, the structure of 3D big data is layered, block storage and two-level memory cache framework are implemented, concurrent access and different resolution streaming display are supported. The octree distributed storage framework, which consists of data access client, metadata service and sub-data block access service, shields the background redundancy or backup storage of data blocks in hundreds of computers based on octree segmentation. The distributed file access interface, which is almost identical to the single machine file system interface, is implemented by the task execution client, the task management service and the earthquake oriented, The distributed big data analysis framework composed of a series of professional interpretation algorithms for digital core or logging data is used to realize the parallel programming model of "divide-and-conquer" by using the characteristics of octree block storage. The complexity of parallel programming model implementation is significantly reduced. The final purpose of PetroV software architecture design and its derivative series of professional software versions under big data is to promote the deep combination of application orientation and new technology development, to continuously pursue excellence, and to enhance the ability of independent innovation.
【作者单位】：中国石化石油勘探开发研究院;
【基金】：国家页岩油重大专项(2017ZX05049001-007) 中科院A类战略性先导科技专项(XDAXX010405)联合资助
【分类号】：P618.13;P631.4

【参考文献】