基于移动ng4.0后台驱动平台的大数据同步的分析与研究

发布时间：2018-06-04 05:07

本文选题：大数据 + CRM系统　；参考：《湖南师范大学》2015年硕士论文

【摘要】：伴随着大数据时代的到来,数据已经成为了各大企业一笔重大的财富,怎样将数据有效的管理起来便成了一个亟待解决的问题,将数据拆分出来保存在不同的地域和不同的存储空间上,是一个有效解决数据管理问题的方案;于是对大数据同步的研究,便成为了一个热门的研究方向。数据同步的研究在系统容灾、智慧校园、智慧农业、智能物流等方面都得到了很多的应用,具有重要的应用意义。本文以“移动ng4.0项目改造工程”中的数据同步为背景,对目前移动NGBOSS系统中的CRM和BILLING两大平台进行了详细的分析,针对移动业务运营支撑系统中CRM和BILLING系统之间数据同步存在的同步速度慢,出现误差频次高,需要人工处理概率大等问题提出了新的解决方案,本文的主要工作有以下两点:针对目前业务运营支撑系统在生成同步文件的过程中没有规范同步文件的大小,导致数据在同步过程中出现同步消耗时间长,甚至出现同步进程僵死,需要人工处理的问题,本文在同步流程中设计了一个自定义数据开关,其主要思想是:将数据开关以参数的形式定义在XML文件中,用来规定每个同步文件能容纳的数据阀值,CRM主机根据开关参数捞取同步数据。实验数据表明这种方式能控制同步文件的大小,解决了由于同步文件过大导致同步数据失败,进而影响整个系统运营的问题。针对同步文件中存在重复数据,数据记录排列杂乱无序,浪费磁盘空间,消耗网络资源多,影响同步效率的问题,本文提出了对同步文件进行序列化处理的解决方案,其主要思想是:CRM侧首先生成同步临时文件,通过JAVA程序交叉调用Linux命令方法,对同步临时文件进行合并,删除文件中的重复记录并重新编排同步文件序列,这种方法能有效的删除同步文件中的冗余数据,规范同步文件中的数据序列,有利于节省主机磁盘空间,减少文件在同步过程中对网络资源的消耗,提高了数据入库效率。
[Abstract]:With the arrival of the big data era, data has become a major wealth of large enterprises, how to effectively manage the data has become a problem to be solved. Splitting data into different regions and different storage space is an effective solution to the problem of data management, so the research on big data synchronization has become a hot research direction. The research of data synchronization has been applied in many fields, such as system disaster recovery, intelligent campus, intelligent agriculture, intelligent logistics and so on. Based on the background of data synchronization in "Mobile ng4.0 Project Reconstruction Project", this paper makes a detailed analysis of the two major platforms of CRM and BILLING in the current mobile NGBOSS system. Aiming at the problems of low synchronization speed, high error frequency and high probability of manual processing between CRM and BILLING systems in mobile business operation support system, a new solution is proposed. The main work of this paper is as follows: at present, the business operation support system does not standardize the size of the synchronization file in the process of generating synchronization files, which results in a long time of synchronization in the process of data synchronization. In this paper, a custom data switch is designed in the synchronization process. The main idea is to define the data switch in the XML file in the form of parameters. It is used to specify the data threshold for each synchronization file, and the CRM mainframe acquires synchronous data according to the switch parameters. The experimental data show that this method can control the size of the synchronization file and solve the problem of the failure of synchronization data caused by the too large synchronous file and then affect the operation of the whole system. Aiming at the problems of duplicated data in synchronous files, disorderly arrangement of data records, waste of disk space, consumption of network resources and influence of synchronization efficiency, this paper proposes a solution to serialize synchronous files. The main idea is to first generate synchronous temporary files on the: CRM side, cross call the Linux command method through JAVA program, merge the synchronous temporary files, delete the duplicate records in the files and rearrange the sequence of synchronous files. This method can effectively delete redundant data in synchronous files, standardize the data sequence in synchronous files, save host disk space, reduce the consumption of network resources in the synchronization process of files, and improve the efficiency of data storage.
【学位授予单位】：湖南师范大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TP311.13

【参考文献】