基于内存计算技术的企业海量财务数据实时分析

发布时间：2018-01-18 00:19

本文关键词：基于内存计算技术的企业海量财务数据实时分析　出处：《复旦大学》2013年硕士论文　论文类型：学位论文

【摘要】：存储在企业内部计算机系统之中的日益增长的财务数据暗藏着诸多有价值的信息。在不断变化的数据分析这一需求下,当这些待分析的数据的量大到一定程度之后,如何实施数据分析操作并实时获得结果正日益成为一个挑战。若使用传统方法,分析操作被发起之后,往往需要等待一段不短的时间,从几十分钟到几天不等,待系统处理结束之后才得到结果。这样实时性需求无法得到满足。为了缩短处理时间,有些系统采取预生成预先定义维度的物化的中间汇总结果这一技术,这样只能支持预先定义维度的相关信息的快速获取,无法满足灵活多变地任选维度进行数据分析的需求。内存计算技术是目前比较热门的技术话题。它采用列方式将数据存储在磁盘中,并且将数据按列压缩。运行时数据全部加载到计算机主存中。在数据操作过程中,一旦数据被需要时立即从主存加载到中央处理器缓存中被处理,相对于传统方法中数据被需要时,先从磁盘加载到主存,然后再从主存加载到中央处理器缓存,大大减少了中央处理器因等待数据而空闲的时间。同时它充分利用计算机中央处理器的多内核,并行处理数据操作。这样大大缩短一次数据分析操作的所需时间。除了从磁盘到中央处理器缓存的数据加载过程这一影响性能的瓶颈之外,另一个瓶颈是：在当前流行的三层企业系统架构体系下,数据总是需要在数据层和逻辑层之间双向传递多次,当数据量达到海量时,这对性能的消耗相当可观。内存计算技术提供在数据层进行内存计算的能力,逻辑层和数据层之间的界限被去除,大大减少大数据传递所带来的性能损耗。本文在分析内存计算技术及其实现框架之后,提出了一个两层架构的企业海量财务数据实时分析系统。以两层架构中的数据层为重点,首先详细阐述系统要分析的数据对象,接着明确系统的设计原则,确定系统功能与界限,提出系统设计方案,然后详细讨论系统实现的主要部分：数据层模型视图,最后通过比较测试证明内存计算技术在企业海量财务数据实时分析领域的价值。
[Abstract]:The growing financial data stored in the internal computer system of the enterprise contains a lot of valuable information. When the amount of data to be analyzed is large enough, how to implement the data analysis operation and obtain the results in real time is becoming a challenge day by day. If the traditional method is used, the analysis operation will be initiated. It is often necessary to wait for a long time, ranging from dozens of minutes to a few days, until the system process is finished before the results can be obtained. Thus, the real-time requirements can not be met, in order to shorten the processing time. Some systems adopt the technique of pre-generating the intermediate aggregate results of the materialization of predefined dimensions, which can only support the rapid acquisition of relevant information of the predefined dimensions. Can not meet the flexible optional dimensions for data analysis. Memory computing technology is a hot topic at present. It uses column method to store data on disk. And the data is compressed by column. The runtime data is loaded into the main memory of the computer.; during the data operation, the data is loaded from the main memory into the central processing unit cache as soon as the data is needed. When data is needed relative to traditional methods, it is loaded from disk to main memory and then from main memory to central processor cache. It greatly reduces the central processor's idle time due to waiting for data, and makes full use of the multi-kernel of the computer's central processor. Parallel processing of data operations. This greatly reduces the time required for a data analysis operation. In addition to the performance bottleneck of the data loading process from disk to CPU cache, another bottleneck is: under the current popular three-tier enterprise system architecture. Data always needs to be transferred between the data layer and the logic layer for many times. When the amount of data reaches a large amount of data, the performance consumption is considerable. Memory computing technology provides the ability to compute memory in the data layer. The boundary between logical layer and data layer is removed, greatly reducing the performance loss caused by big data transfer. After analyzing the memory computing technology and its implementation framework, this paper puts forward a real-time analysis system of enterprise magnanimity financial data with a two-tier architecture, focusing on the data layer in the two-tier architecture. Firstly, the data objects to be analyzed are described in detail, then the design principles of the system are defined, the functions and limits of the system are determined, and the design scheme of the system is put forward. Then the main part of the system implementation: data layer model view is discussed in detail. Finally, the value of memory computing technology in the field of real-time analysis of massive financial data is proved by comparison test.
【学位授予单位】：复旦大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】