CLAIMS系统中实时数据注入的研究与实现

发布时间：2018-03-01 08:47

本文关键词： 实时数据注入事务分布式数据库 CLAIMS　出处：《华东师范大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着互联网的发展,面对海量数据的存储和管理问题,金融等行业开始采用分布式数据库系统以提升系统的数据处理能力和可扩展性。为减少因数据注入延时和分析延时带来的数据价值下降,金融等行业要求其系统支持实时的数据注入和查询;同时也需要事务性的注入和查询以保证金融数据的正确性和一致性。而传统分布式环境下的事务处理大多基于锁和两阶段提交,无法实现高通量实时数据注入,另一方面NoSQL分布式系统不支持事务型数据注入。CLAIMS系统是开源的分布式内存OLAP系统,支持实时数据查询,但需要功能扩展以实现事务性实时数据注入。本文针对CLAIMS系统,设计基于元数据的事务并发控制机制,进而提出分布式事务型实时数据注入框架,并通过大量实验证明其满足金融等领域的需求。本文的主要贡献如下:1.针对CLAIMS系统设计基于元数据的事务并发控制机制,通过逻辑与物理相分离来达到控制元数据即控制事务数据的目的,原子性并发控制元数据以隔离写事务,利用事务快照以实现读写分离,支持强一致性。2.在实现分布式事务管理器的基础上,本文设计事务型分布式实时数据注入引擎,广泛使用无锁结构提供高吞吐量和低延时的实时数据注入,通过事务保证注入过程的ACID特性,且具有较高的可扩展性。3.在分布式OLAP系统上实现事务型实时数据注入框架,并通过一系列功能测试,与VoltDB进行性能对比测试,证明本文设计的框架在满足事务特性的同时,能提供较高通量低时延的实时数据注入服务。本文提出的基于元数据的事务并发控制方式具有一定的创新性和学术价值,设计实现的实时数据注入基本满足工业生产环境的需求,具有一定的实践价值。本文的工作对我国金融行业的分布式数据实时注入具有借鉴意义。
[Abstract]:With the development of the Internet, facing the problem of massive data storage and management, In order to reduce the data value caused by data injection delay and analysis delay, finance and other industries began to adopt distributed database system to improve the data processing ability and extensibility of the system. Finance and other industries require their systems to support real-time data injection and query; At the same time, transactional injection and query are also needed to ensure the correctness and consistency of financial data. In the traditional distributed environment, most transactions are based on locks and two-phase commit, so high-throughput real-time data injection can not be realized. On the other hand, NoSQL distributed system does not support transactional data injection. CLAIMS is an open source distributed memory OLAP system, which supports real-time data query, but it needs function expansion to realize transactional real-time data injection. The transaction concurrency control mechanism based on metadata is designed, and a distributed transactional real-time data injection framework is proposed. The main contributions of this paper are as follows: 1. The transaction concurrency control mechanism based on metadata is designed for CLAIMS system. The purpose of controlling metadata, i.e. controlling transaction data, is achieved by separating logic from physics. Atomic concurrent control metadata is used to isolate write transactions, and transaction snapshots are used to realize the separation of reading and writing. Based on the implementation of distributed transaction manager, this paper designs a transactional distributed real-time data injection engine, which provides high throughput and low latency real-time data injection. The transactional real-time data injection framework is implemented on the distributed OLAP system, and a series of functional tests are carried out to compare the performance of the framework with that of VoltDB. It is proved that the proposed framework can provide real-time data injection services with high throughput and low delay while satisfying the transaction characteristics. The proposed transaction concurrency control method based on metadata has some innovative and academic value. The design and implementation of real-time data injection basically meet the needs of the industrial production environment and have certain practical value. The work of this paper has reference significance for the distributed data real-time injection in the financial industry of our country.
【学位授予单位】：华东师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【参考文献】