一种基于OpenTSDB的海量实时数据存储系统

发布时间：2018-03-07 23:08

本文选题：分布式系统　切入点：OpenTSDB　出处：《华南理工大学》2016年硕士论文　论文类型：学位论文

【摘要】：海量数据的存储与查询是进行大数据分析的先决条件,如何高效灵活的存取海量数据已成为当前业内的研究热点。华南理工大学能耗数据分析平台采用传统的关系型数据库作为其存储支撑,但受限于其理论模型与体系结构,很难在性能和扩展性方面满足大数据情景下的数据需求。非关系型数据库可以较好的解决这些问题,但其功能往往较为简单,在复杂查询和事务管理等方面支持有限,并且缺乏标准化的查询语言或接口,难以与基于SQL的查询逻辑兼容,技术迁移难度大。为了解决上述问题,本文对能耗分析平台内实时数据的特征进行了详细分析,并在广泛技术调研的基础上,整合关系型数据库与非关系型数据库的各自优势,设计并实现了一套基于OpenTSDB的海量实时数据存储系统。该系统的核心设计思想是:构建由关系数据库和非关系数据库OpenTSDB组成的异构数据库集群,关系数据库中存储关系特征强的、存在事务或复杂查询需求的数据;OpenTSDB则用于支撑海量实时数据的存储。在数据持久层模块上,本文创新性的提出了一种基于面向切面编程的设计方案,借助Spring AOP增强Mybatis这一关系型数据库数据持久化框架的查询方法,在一次接口调用过程中同时实现两类数据库的查询,用关系数据库查询结果驱动OpenTSDB的查询,并最终按照接口定义返回规范的查询结果。这一设计方案无需对相关框架或数据库的代码进行修改,实现了各个模块之间的松耦合,并且不会对上层的业务逻辑产生影响,具有良好的兼容性,极大的降低了技术迁移的难度。本文还提出了一种基于冗余存储的顺序读取优化设计方案,以额外的存储空间为代价换取了使用不同字段进行顺序读取效率提升。华南理工大学能耗数据分析平台上进行的一系列测试显示基于OpenTSDB的海量实时数据存储系统具备完善的功能并有着良好的随机/顺序访问性能。
[Abstract]:Storing and querying massive data is a prerequisite for big data's analysis. How to access mass data efficiently and flexibly has become a research hotspot in the industry. The energy consumption data analysis platform of South China University of Technology uses the traditional relational database as its storage support, but is limited by its theoretical model and architecture. It is difficult to meet the data requirements in big data's situation in terms of performance and extensibility. Non-relational databases can solve these problems better, but their functions are often relatively simple, and their support for complex queries and transaction management is limited. And lack of standardized query language or interface, it is difficult to be compatible with query logic based on SQL, technology migration is difficult. In order to solve the above problems, the characteristics of real-time data in the energy consumption analysis platform are analyzed in detail. And on the basis of extensive technical investigation, integrating the respective advantages of relational database and non-relational database, A massive real-time data storage system based on OpenTSDB is designed and implemented. The core of the system is to construct a heterogeneous database cluster composed of relational database and non-relational database OpenTSDB. OpenTSDB, which has transaction or complex query requirements, is used to support the storage of massive real-time data. In the module of data persistence layer, this paper proposes an innovative design scheme based on section-oriented programming. With the help of Spring AOP to enhance the query method of Mybatis, which is the framework of relational database data persistence, the query of two kinds of databases is realized simultaneously in the process of single interface call, and the query of OpenTSDB is driven by the query result of relational database. Finally, the standard query result is returned according to the interface definition. This design scheme does not need to modify the code of the related framework or database, realizes the loose coupling between the various modules, and does not have an impact on the business logic of the upper layer. It has good compatibility and greatly reduces the difficulty of technology migration. This paper also proposes an optimized design scheme of sequential reading based on redundant storage. A series of tests conducted on the platform of energy consumption data analysis of South China University of Science and Technology show that the massive real-time data storage system based on OpenTSDB is based on the results of a series of tests conducted on the platform of energy consumption data analysis of South China University of Technology. Has the perfect function and has the good random / sequential access performance.
【学位授予单位】：华南理工大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP333

【相似文献】