云产品数据多维分析系统的设计与实现

发布时间：2018-03-16 03:03

本文选题：多维分析　切入点：联机分析处理　出处：《北京交通大学》2017年硕士论文　论文类型：学位论文

【摘要】：近年来随着云计算行业的快速发展,以云服务为主营业务的互联网公司也得到了快速的成长。目前笔者所在公司已有近13条云产品业务线,而各业务线每天会产生庞大且分散的业务数据,其中日志数据已达到了 PB级别。如何对各云产品进行统一的业务分析,及时发现各产品线运营中存在的问题和预测市场需求的变化趋势,是公司目前所面临的难题。针对这一问题,本文旨在接入各产品线业务数据搭建一个统一的、快速响应的云产品数据报表平台,提供多维度深层次的报表查询、数据统计、数据分析、数据预测等功能。在云产品数据多维分析系统的构建过程中,首先,笔者完成了数据主题与数据维度的统计分析,并且参与了多维分析系统的界面原型设计。其次,笔者参与了联机分析处理OLAP(On-line Analytical Processing)引擎调研和日志处理方案的设计工作。然后,在系统设计与实现阶段,笔者独立设计实现了多维分析子系统、Trainer 子系统和日志转 ORC(Optimized Row Column File)子系统中的 CDN(Content Delivery Network)点播日志处理与视频直播日志处理,并完成了数据模型与Cube的创建工作。在测试阶段,笔者独立完成了所负责相关子系统的功能测试。本文所设计的数据多维分析系统基于Apache Kylin的OLAP引擎进行构建,采用预计算的方式预先计算Cube立方体从而实现高响应的数据多维分析。多维分析子系统根据复杂的业务逻辑动态的构建SQL,并对查询后的数据进一步加工处理从而更有效的帮助决策。Trainer子系统负责数据的定期同步以及Cube的自动构建,为多维分析子系统提供数据支撑,并保证数据的一致性。日志转ORC子系统负责对各产品线的日志数据进行ETL(Extract-Transform-Load)处理,该子系统采用SparkSQL对日志数据进行并行计算,极大提高了处理速度,并采用更加高效的压缩格式ORC来存储处理后的数据,提高了后续的数据处理性能。目前,该云产品数据多维分析系统完成了一期的开发与测试工作,并且正处于试运行阶段,各系统的功能运行稳定,达到了预期目标。
[Abstract]:With the rapid development of cloud computing industry in recent years, Internet companies with cloud service as their main business have also been growing rapidly. And every business line produces huge and scattered business data every day, in which log data have reached PB level. It is a difficult problem for the company to find out in time the problems existing in the operation of each product line and to predict the changing trend of market demand. In view of this problem, the purpose of this paper is to access the business data of each product line and build a unified one. The rapid response cloud product data report platform, provides the multi-dimensional deep level report query, the data statistics, the data analysis, the data forecast and so on function. In the cloud product data multidimensional analysis system construction process, first, The author has completed the statistical analysis of the data subject and data dimension, and participated in the design of the interface prototype of the multidimensional analysis system. Secondly, the author has participated in the design of the on-line analytical processing OLAP(On-line Analytical processing engine investigation and log processing scheme. In the stage of system design and implementation, the author designs and implements the CDN(Content Delivery Network) log processing and live video log processing in the multidimensional analysis subsystem and the CDN(Content Delivery network network subsystem. In the testing stage, the author independently completed the functional testing of the related subsystems. The data multidimensional analysis system designed in this paper is based on the OLAP engine of Apache Kylin. The precomputed Cube cube is used to realize the multidimensional analysis of highly responsive data. The multidimensional analysis subsystem constructs SQL dynamically according to the complex business logic, and further processes the queried data. A more effective help decision. Trainer subsystem is responsible for the periodic synchronization of data and the automatic construction of Cube, It provides data support for multidimensional analysis subsystem and ensures consistency of data. Log to log ORC subsystem is responsible for processing log data of each product line by ETLL Extract-Transform-Load. This subsystem uses SparkSQL to compute log data in parallel. The processing speed is greatly improved, and more efficient compression format ORC is adopted to store the processed data, and the subsequent data processing performance is improved. At present, the cloud product data multidimensional analysis system has completed the first stage of development and testing. And is in the trial operation stage, each system function runs stably, has achieved the anticipated goal.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.09

【相似文献】