基于多维联机分析的分布式审计系统的设计与实现

发布时间：2018-06-21 01:08

本文选题：联机分析 + 数据审计　；参考：《东南大学》2016年硕士论文

【摘要】：随着云时代的到来,海量结构化数据的审计正成为伴随着电子商务、物联网等新兴互联网领域而逐渐兴起的热点问题,历史数据的审计的目的在于尽可能地通过日志文件等记录数据,实现对威胁或潜在威胁的发现和防范。其中的难点在于如何实现其汇总数据查询的即席性、联机性和交互性。在数据审计领域,对数据仓库的审计主要有两种方法,即账表还原法和基础数据核查法,然而问题是,现阶段,在数据量逐渐增大的压力下,不管何种审计方法都无法在审计速度、审计数据存储方面取得令人满意的效果,审计任务完成不及时、审计数据无法妥善存储都会给企业和政府带来不可估量的损失。本文基于数据立方体格模型及分布式技术,设计一套高效率的基于分布式和联机分析技术的审计系统,力求做到数据审计的快速高效。本文主要做了如下工作：(1)通过对各种分布式技术的优劣进行比较,选取出适合大数据审计的分布式框架,并将审计数据从传统的数据库或excel移植到类Hadoop MapReduce的通用并行框架Spark上。(2)在分布式平台上对聚集数据进行压缩。首先对分布式存储平台的特点进行分析,选用封闭立方体技术对聚集数据进行压缩,压缩后的数据以封闭立方体的形式存储在HDFS上,以供审计查询。(3)得到压缩的封闭立方体后,结合Spark编程模型RDD的特点在分布式系统上对其进行查询,查询方式与已有的查询方式不同,因为要兼顾分布式系统的特点和查询效率,做到快速准确。(4)整合前面的技术,设计出合适的基于分布式系统的审计架构。解决了如下两个问题,1)海量数据的存储和聚集数据的查询。2)审计规则在大数据平台的应用。即结合封闭立方体技术、分布式技术、审计技术,设计一套实用的大数据审计系统,并以实验的方式对其功能进行验证。
[Abstract]:With the arrival of the cloud era, the audit of massive structured data is becoming a hot issue gradually rising along with the electronic commerce, the Internet of things and other emerging Internet fields. The purpose of the audit of historical data is to realize the discovery and prevention of threats or potential threats by recording data such as log files as much as possible. The difficulty lies in how to realize the ad hoc, online and interactivity of its aggregate data query. In the field of data audit, there are mainly two methods for data warehouse audit, namely, the method of reducing the account sheet and the method of checking the basic data. However, the problem is, at the present stage, under the pressure of increasing data volume, No matter what audit method can not be used in audit speed, audit data storage can achieve satisfactory results, audit task is not completed in time, and audit data can not be stored properly will bring incalculable losses to enterprises and governments. Based on the data cube lattice model and distributed technology, this paper designs an efficient audit system based on distributed and on-line analysis technology, which aims to achieve fast and high efficiency of data audit. The main work of this paper is as follows: (1) by comparing the advantages and disadvantages of various distributed technologies, we select a distributed framework suitable for big data audit. And the audit data is transplanted from traditional database or excel to the general parallel framework of Hadoop MapReduce, named Spark. 2) the aggregate data is compressed on distributed platform. Firstly, the characteristics of distributed storage platform are analyzed, and the aggregate data is compressed by using closed cube technology. The compressed data is stored on HDFS in the form of closed cube, so that the compressed closed cube can be obtained by audit query. According to the characteristics of Spark programming model RDD, it is queried on the distributed system. The query way is different from the existing query way, because the characteristics and query efficiency of the distributed system should be taken into account, so that the former technology can be integrated quickly and accurately. An appropriate audit architecture based on distributed system is designed. The following two problems are solved: 1) the application of the audit rules to the big data platform for the storage of massive data and the query of aggregate data. Combining the closed cube technology, distributed technology and audit technology, a set of practical big data audit system is designed, and its function is verified by experiment.
【学位授予单位】：东南大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】