传统关系型数据库向非关系型数据库迁移算法研究

发布时间：2018-10-11 12:52

【摘要】：近年来,随着物联网技术的应用和“互联网+”概念的提出,互联网中的数据量呈出指数式的增长从而由信息时代进入大数据时代。大数据时代的数据不仅在数据量上剧增还在结构、分布、价值密度及维度等方面有所改变,这些数据在并行化快速处理上的要求需要具有半结构化及非结构化数据存储、高扩展性和高读写行能的非关系型数据库及分布式计算模式来满足。传统关系型数据库无法满足处理大数据的要求,但是由于其广泛应用而在历史上积累了大量的数据,为了充分利用这些数据继续为用户提供服务、为企业创造更多的价值需要设计一种自动化或半自动化的迁移工具来完成传统关系型数据库向非关系型数据库的转换。另外,由于非关系型数据库与传统关系型数据库的使用接口有着本质上的区别,为了使迁移后原有业务应用能够适应新的数据存储模型推动向非关系型数据库的平滑过渡需要将非关系型数据库的原生操作接口进行封装并通过设计一种SQL操作中间件来实现这个转变。本文通过对数据库存储结构、大数据相关技术和HBase的操作接口的研究,设计实现了数据库迁移工具并对HBase的操作接口进行了封装。首先,对数据存储和迁移的相关技术进行了研究,深入地分析了传统关系型数据库的存储原理和设计规则、HBase数据库的架构和存储模式、Hadoop生态系统以及数据迁移设计的相关模型和技术。其次,针对传统关系型数据库的元数据进行深入分析提出并设计实现了表模式和数据由MySQL向HBase迁移的算法。然后,对HBase的原生API进行深入分析,利用SQL解析和Filter条件过滤并结合传统关系型数据库的操作习惯对其进行了封装。最后,通过设计实验进行性能测试,分析结果表明本文所实现的迁移工具和操作中间件能够快速有效地完成历史数据的迁移工作并在以后的使用过程中满足查询需求,为数据存储从关系型数据库向非关系型数据库的过渡提供了保障。
[Abstract]:In recent years, with the application of the Internet of things technology and the concept of "Internet", the amount of data in the Internet has increased exponentially, thus from the information age to big data era. The data of big data era not only increased sharply in data volume, but also changed in structure, distribution, value density and dimension, etc. These data need to be stored in semi-structured and unstructured data in parallelization and fast processing. Non-relational databases and distributed computing models with high scalability and high read-write performance are satisfied. The traditional relational database can not meet the requirement of processing big data, but because of its extensive application, it has accumulated a large amount of data in history. In order to make full use of this data, it continues to provide services to users. To create more value for enterprises, we need to design an automatic or semi-automatic migration tool to complete the transformation from traditional relational database to non-relational database. In addition, because the interface between non-relational database and traditional relational database is essentially different, In order to adapt the new data storage model to the smooth transition to non-relational database, the native operation interface of non-relational database is encapsulated and a SQL operation is designed. Middleware to achieve this transformation. Through the research of database storage structure, big data technology and HBase operation interface, this paper designs and implements the database migration tool and encapsulates the operation interface of HBase. Firstly, the related technologies of data storage and migration are studied. The storage principle and design rules of traditional relational database, the structure and storage mode of HBase database, the model and technology of Hadoop ecosystem and data migration design are analyzed. Secondly, an algorithm of table schema and data migration from MySQL to HBase is proposed and implemented by analyzing the metadata of traditional relational database. Then, the native API of HBase is deeply analyzed and encapsulated by using SQL parsing and Filter conditional filtering and combining with the operation habits of traditional relational database. Finally, through designing experiments to test the performance, the analysis results show that the migration tools and operation middleware implemented in this paper can quickly and effectively complete the migration of historical data and meet the query requirements in the later use process. It provides a guarantee for the transition of data storage from relational database to non-relational database.
【学位授予单位】：华侨大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【参考文献】