面向城市基础设施智慧管养的大数据智能融合方法

发布时间：2018-01-19 18:54

本文关键词： 大数据抽取-转换-加载数据融合数据仓库城市基础设施管养　出处：《计算机应用》2017年10期 　论文类型：期刊论文

【摘要】：针对运维大数据维度高、形式多样化和变化迅速等特性,为提高数据融合效率以及平台的数据统计和决策分析性能,降低抽取-转换-加载(ETL)执行时间开销和数据中心负担,面向智慧管养需求提出一种多层次任务调度(MTS)ETL框架(MTS-ETL)。首先,将数据仓库分为数据临时区、数据仓储区、数据分类区和数据分析区,并根据所分区域将完整的ETL过程划分为4个层次的ETL任务调度环节,同时设计了多频率ETL运行调度以及顺序和非顺序两种ETL工作模式;接着,基于MTS-ETL框架的非顺序工作模式进行数据融合的概念建模、逻辑建模和物理建模;最后,利用Pentaho Data Integration设计ETL转换模块和工作模块以实现数据融合方法。在交通流量数据融合实验中,该方法融合136 754条数据的时间仅为28.4 s;在千量级的数据融合实验中比传统ETL方法的总平均执行时间降低了6.51%;报表分析结果表明其在融合400万条数据时依然能保证ETL过程的可靠性。所提方法能够有效融合运维大数据,提高平台统计分析性能,并维持ETL执行时间开销在较低水平。
[Abstract]:In order to improve the efficiency of data fusion and the performance of data statistics and decision analysis, big data has the characteristics of high dimension, diversified forms and rapid change. Reduce the decimation-transform-load ETL) execution overhead and data center burden. To meet the requirement of intelligent management, a multi-level task scheduling framework (MTS-ETL) is proposed. Firstly, the data warehouse is divided into data temporary area and data storage area. The whole ETL process is divided into four levels of ETL task scheduling according to the data classification area and data analysis area. At the same time, the multi-frequency ETL scheduling and two ETL modes, sequential and non-sequential, are designed. Then, the concept modeling, logical modeling and physical modeling of data fusion are carried out based on the non-sequential work pattern of MTS-ETL framework. Finally, using Pentaho Data Integration to design ETL conversion module and work module to realize data fusion method. In traffic flow data fusion experiment. The time of fusion of 136 754 data by this method is only 28.4 s; In the data fusion experiment, the average execution time of the traditional ETL method is 6.51 less than that of the traditional ETL method. The results of report analysis show that the method can still ensure the reliability of ETL process when 4 million pieces of data are fused. The proposed method can effectively fuse the operation and maintenance big data and improve the performance of statistical analysis of the platform. And maintain ETL execution time at a lower level.
【作者单位】：上海大学悉尼工商学院;上海大学—上海城建建筑产业化研究中心;
【基金】：上海市城乡建设和交通委员会建管项目(2014-009-002) 上海市科委重点项目(13511504803) 上海市国资委重大科研项目(2014008)~~
【分类号】：TP311.13
【正文快照】： 0引言城市基础设施管养是指充分利用信息化和大数据技术对城市道路、桥梁以及隧道等各类市政设施进行智慧化管理和养护,它涵盖了交通建设运营、设施设备管理、项目资产巡检以及养护合同收益等多项业务。随着智慧城市建设的推进,城市基础设施管养精细化的需求日渐突出,海量历史

【相似文献】