基于Spark Streaming的实时能耗分项计量系统

发布时间：2018-03-02 19:48

本文选题：流式计算　切入点：能耗分项计量　出处：《计算机应用》2017年04期 　论文类型：期刊论文

【摘要】：能耗分项计量能够准确、及时、有效地发现能源使用问题,形成和实现最有效的节能措施。能耗分项计量系统需要对各项能源使用量在不同粒度上进行统计,既有实时性的需求,又需要涉及到聚合、去重、连接等较为复杂的统计需求。由于数据产生快、实时性强、数据量大,所以很难统一采集并入库存储后再作处理,这便导致传统的数据处理架构不能满足需求。为此,提出基于Spark Streaming大数据流式技术构建一个实时能耗分项计量系统,对实时能耗分项计量的系统架构和内部结构进行了详细介绍,并通过实验数据分析了系统的实时数据处理能力。与传统架构不同,实时能耗分项计量系统在数据流动的过程中实时地进行捕捉和处理,一方面把捕捉到的异常信息及时报警到前端,同时把分类分项统计处理的结果保存到数据库,以便进行离线分析和数据挖掘,能有效地解决上述数据处理过程中遇到的问题。
[Abstract]:Energy consumption itemized metering can accurately, timely and effectively find energy use problems, and form and realize the most effective energy saving measures. There are not only real-time requirements, but also complex statistical requirements, such as aggregation, heavy removal, connection and so on. Because of fast data generation, strong real-time performance and large amount of data, it is very difficult to collect and store the data in a unified way and then process it. As a result, the traditional data processing architecture can not meet the demand. Therefore, a real-time energy consumption sub-measurement system based on Spark Streaming large data flow technology is proposed. This paper introduces the system architecture and internal structure of real time energy consumption measurement in detail, and analyzes the real time data processing ability of the system through experimental data. The real-time energy consumption itemized measurement system captures and processes the data flow in real time. On the one hand, the abnormal information caught is alerted to the front end in time, and the results of the statistical processing of the classified items are saved to the database at the same time. In order to do offline analysis and data mining, can effectively solve the problems encountered in the process of data processing.
【作者单位】：成都五舟汉云科技有限公司;成都信息工程大学信息安全工程学院;
【分类号】：TP311.13

【参考文献】