基于日志解析的数据库海量数据同步系统的研究与实现

发布时间：2018-04-06 07:37

本文选题：数据库　切入点：数据同步　出处：《华南理工大学》2016年硕士论文

【摘要】：随着我国电子商务和金融行业的迅猛发展,数据的存储和交易量已经进入海量数据时代。与此同时,数据库已经从集中式走向分布式,数据同时存在于位置分散的不同服务器上,保证企业系统的高可靠运行。这又引入一个关键的问题,如何保持不同分布式节点上的数据一致性。基于这样的时代和技术背景,本文提出了基于日志解析的数据库海量数据同步系统的解决方案,设计并实现了数据同步系统的原型。本文首先调查了各行业数据库异常所带来的影响,阐述了数据同步建设的必要性和急迫性,并简要分析了国内外的技术现状和应用情况。针对数据库同步中的几个关键问题,总结了现有的几种主流数据同步方法和增量数据的捕获方法,并分析比较了各种方法的优劣。根据这些技术选择出系统可行的技术路线,采用性能更优的日志解析法来还原SQL操作,以及采用时效性更高的文件过滤驱动技术实时捕获日志文件的增量数据,并使用了市场占有率最高的Oracle数据库作为研究对象。其次,紧紧围绕如何从尚未公开的Oracle数据库二进制日志文件中提取出SQL操作信息。为此做了大量实验,并依据实验数据得出日志文件的详细内部结构,识别出日志文件的整体逻辑结构,获得最外层的重做块结构、中间层的重做记录结构和最里层的重做变量向量的结构以及每层结构中包含的内容。并最终确定每个原子操作所对应的Redo Change Vector结构,以及该结构中保存的字段信息和行记录相对应的物理数据块的地址信息,根据这些信息重构出SQL语句。接着,在上述实验的基础上,设计了整个数据同步系统的原型,并对系统的实现与验证进行了详细的论述。以图示的形式描述了系统的整体框架和各功能子系统之间的逻辑结构,该系统包括4个部分:日志监控子系统、日志解析子系统、数据传输子系统和数据写入子系统,并对每个子系统的架构设计、处理流程、数据存储和使用的关键技术等进行了详细的说明。在系统实现与验证部分,对各个子系统和功能模块所包含的函数的实现进行了详细描述,并给出了关键代码。同时对该数据同步系统进行了功能测试、性能测试和兼容性测试,验证了系统的可行性、可靠性和可扩展性。最后,对文章内容和存在不足进行了总结,并对系统中的可改进地方提出了思路和建议。
[Abstract]:With the rapid development of China's e-commerce and financial industry, storage and trading volume data has entered the era of massive data. At the same time, the database has been from centralized to distributed data, and have the different server in scattered locations, ensuring high reliable operation of the enterprise system. It also introduced a key issue, how to keep the data the consistency of different distributed nodes. So the time and technology based on the background, this paper puts forward a solution of database data synchronization system based on log analysis, design and implementation of data synchronization system prototype. This paper investigated the influence of each industry database exception brings, expounds the necessity and urgency of data synchronous construction, and analyzes the technology and application status at home and abroad. Aiming at some key problems of data synchronization, the total. Existing acquisition methods of mainstream data synchronization method and incremental data, and analyzes the advantages and disadvantages of various methods. According to the technology selection of feasible technology system, to restore the SQL operation using the log analysis method with better performance, and the use of more efficient file filter driver technology to capture the incremental data real-time log file the highest market share and use the Oracle database as the research object. Secondly, focus on how to extract SQL information from the Oracle database binary log file has not been made public. To do a lot of experiments, and according to the experimental data obtained with the internal structure of the log files, identify the overall logical structure of the log file. Get to redo the outermost block structure, the structure of variable vector redo redo records structure of the middle layer and the innermost layer and each layer The contents of Redo Change Vector. And ultimately determine the structure corresponding to each atomic operation, and physical data stored in the structure in the field of information and records the corresponding block address information, according to the information to reconstruct the SQL statement. Then, on the basis of above experiments, the design of the entire data synchronization system prototype, and the implementation and verification of the system are discussed in detail. A pictorial form describes the logical structure between the overall framework of the system and the function of each subsystem, the system includes 4 parts: the log monitoring subsystem, log analysis subsystem, data transmission subsystem and data writing system, and the each subsystem architecture design, data storage and processing, the key technologies used in detail. Verification and implementation of the system, which are included in each subsystem and function module The function realization is described in detail, and gives the key code. At the same time, the data synchronization system for functional testing, performance testing and compatibility testing, verify the feasibility of the system, reliability and scalability. Finally, the shortcomings of the existing content and are summarized, and the system can be improved is proposed some ideas and suggestions.

【学位授予单位】：华南理工大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】