人社数据实时采集与异构存储的设计与实现

发布时间：2018-05-18 13:49

本文选题：人社一体化 + 数据异构同步　；参考：《山东大学》2017年硕士论文

【摘要】：随着互联网等信息技术的飞速发展,人力资源与社会保障业务办理方式的改变,办事流程的简化以及处理效率的提高成为了必然。而人力资源和社会保障业务涵盖社会保险、公共就业、劳动关系、人事人才等诸多领域,涉及到的面非常广,伴随而来的业务量也非常大,这使得业务办理流程十分复杂,进而导致效率低下。所以,为了适应时代发展,给人们提供高效便捷的人社业务服务,人力资源和社会保障局提出了人社一体化建设的方针政策。人社一体化建设以人社数据省集中一体化建设为基础;而人社数据省集中一体化建设的过程中存在数据共享、业务协同、生产交换公共服务间的数据同步、服务交换等需求,需要实现人社领域生产区、交换区、公共服务区内部及之间的海量数据的高效同步复制和增量同步转换等目标。基于上述的背景,本文充分考虑了人社数据省集中一体化建设过程中对于人社业务数据在实时采集以及异构存储等方面的各种需求,针对其开发了人社大数据集成平台中的数据异构同步功能子系统。该功能子系统包含了异构复制配置,转换规则管理,数据库和服务资源管理以及转换流程配置等功能模块。人社数据异构同步功能系统允许用户根据业务需求选择需要进行增量数据异构同步的数据库以及其中相应的数据表。同时,系统为用户提供了转换规则开发接口、开发规范,以及相应的开发示例;用户可根据自身转换需求自定义对应的转换逻辑,并将开发的转换逻辑通过系统上传,以便在数据库异构同步过程中进行调用。而且,该系统可以将源数据库某个表的增量变更记录通过调用不同的转换逻辑同时进行不同的格式和内容的转换,并在转换完成后成功应用到一个或多个目标端。最后,系统为数据异构同步过程提供了相应的监控功能,以保证数据异构同步过程可查可控,进而实现了系统的可用性稳定性和异构同步的数据一致性。本论文从系统的需求定位、概要分析、详细设计以及实现测试等方面进行了全面详细的叙述。系统采用B/S架构格式,以基于OSGI规范的Karaf系统作为基础平台,运用插件式开发的方式搭建而成。开发平台选用EclipseIDE;源数据库数据实时采集采用Oracle Streams技术,数据转换逻辑、相关数据库以及相关服务作为资源在系统中发布,并采用Apache Camel技术通过设计相应的数据转换应用路由实现数据的读取、转换和应用过程;系统界面开发采用Vaadin技术;运行平台支持Windows和Linux操作系统。系统经过了充分的测试,在实现了数据异构同步的功能性需求外,也实现了软件易用性可靠性可维护性等非功能需求。
[Abstract]:With the rapid development of information technology such as the Internet, the change of human resources and social security business management, the simplification of work flow and the improvement of processing efficiency become inevitable. Human resources and social security services cover many fields, such as social insurance, public employment, labor relations, personnel, and so on. They involve a wide range of areas, and the accompanying volume of business is also very large. This makes the business process very complex. This leads to inefficiency. Therefore, in order to adapt to the development of the times and provide people with efficient and convenient service, the Bureau of Human Resources and Social Security put forward the principles and policies of the integration of people and society. The integration construction of human society is based on the centralized integration of data province and data province, and there are some requirements such as data sharing, business cooperation, data synchronization between production and exchange of public services, service exchange and so on. It is necessary to achieve the goals of efficient synchronous replication and incremental synchronous conversion of mass data within and between the production area, exchange area and public service area. Based on the above background, this paper gives full consideration to the needs of real-time collection and heterogeneous storage of human society business data in the process of centralized integration of human society data province. The data heterogeneous synchronization subsystem in the human society big data integration platform is developed. The functional subsystem includes heterogeneous replication configuration, transformation rule management, database and service resource management, and transformation process configuration. The human society data heterogeneous synchronization system allows users to select the databases and the corresponding data tables that need to synchronize the increment data according to the needs of the business. At the same time, the system provides the user with the interface of conversion rule development, the development specification, and the corresponding development example, the user can customize the corresponding transformation logic according to his own conversion requirements, and upload the developed transformation logic through the system. In order to make calls in the database heterogeneous synchronization process. Furthermore, the system can transform the incremental change record of a table in the source database into different formats and contents simultaneously by calling different transformation logic, and it can be successfully applied to one or more target terminals after the conversion is completed. Finally, the system provides the corresponding monitoring function for the data heterogeneous synchronization process, to ensure that the data heterogeneous synchronization process can be checked and controllable, thus realizing the usability stability of the system and the data consistency of heterogeneous synchronization. In this paper, the requirements of the system positioning, summary analysis, detailed design and implementation of testing and other aspects of a comprehensive and detailed description. The system adopts the B / S architecture format, takes the Karaf system based on OSGI specification as the basic platform, and uses the plug-in development method to build the system. The development platform is Eclipse IDE, the source database data real-time collection adopts the Oracle Streams technology, the data conversion logic, the related database and the related service are released in the system as the resources. The Apache Camel technology is used to realize the data reading, conversion and application process by designing the corresponding data conversion application route; the system interface development adopts Vaadin technology; the running platform supports the Windows and Linux operating system. In addition to the functional requirements of heterogeneous data synchronization, the system also realizes the non-functional requirements such as ease of use, reliability, maintainability and so on.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【相似文献】