容错存储的电力系统监测数据查询优化技术

发布时间：2017-12-26 18:10

本文关键词：容错存储的电力系统监测数据查询优化技术　出处：《电网技术》2015年11期 　论文类型：期刊论文

【摘要】：针对电力系统监测中大量时间序列数据存储和高效查询的问题,利用云计算框架和HQL(Hive SQL)查询引擎,提出一种容错存储的分级分区查询优化方法。通过副本机制设计电力监测数据容错存储模式,综合运用了HQL查询计划生成、向Map/Reduce的转化和分区剪枝处理,进行了加载和查询优化测试。结果表明,当加载的监测数据记录超过200万条、查询数据记录超过380万条后,HQL的处理性能将远超SQL,数据量越大,优势越明显。分级分区查询测试结果表明,在查询耗时相近的条件下,分区查询的数据量可以扩大2个数量级,且二级分区比一级分区更高效,验证了查询优化技术可有效提高电力系统监测信息查询处理的效率,为大量电力监测数据处理提供了一种查询优化方法。
[Abstract]:In view of the problem of large time series data storage and efficient query in power system monitoring, a cloud computing framework and HQL (Hive SQL) query engine are proposed, and a fault-tolerant storage hierarchical partitioning query optimization method is proposed. The fault-tolerant storage mode of power monitoring data is designed by replica mechanism. HQL query plan generation, transformation to Map/Reduce and partition pruning are applied synthetically. Loading and query optimization tests are carried out. The results show that when the recorded data of the load is more than 2 million, and the query data is more than 3 million 800 thousand, the processing performance of HQL will exceed SQL. The larger the amount of data is, the more obvious the advantage is. Classified query test results show that the time at a similar query, data partition query can be expanded to 2 orders of magnitude, and the two stage partition than a partition is more efficient and verify optimization can effectively improve the power system monitoring information query processing efficiency of query technology, provides a method for query optimization a large number of power monitoring data processing.
【作者单位】：华东交通大学电气工程学院;
【基金】：国家自然科学基金项目(51267005)~~
【分类号】：TM769;TP311.13
【正文快照】： 0引言目前的电力自动化监控系统多采用关系库支撑数据平台[1-2],但越来越多的数据传感设备、移动物联设备、新能源与电动储能设备等的联网接入,使现代电网运行和监测的数据量呈指数倍增[3-6]。关系数据库主要依赖更快速、更昂贵的大型服务器向上纵向扩展,由于数据增长速度远大

【参考文献】