配电网监测数据的分布式Map压缩-查询技术

发布时间：2018-06-15 00:55

  本文选题：配电网 + 大数据　； 参考：《电力自动化设备》2017年12期

【摘要】：针对配电自动化中大量电力监测数据的处理问题,提出了回避聚合操作的配电网监测数据分布式Map压缩-查询新方法。通过将监测数据分布式Map压缩存储,利用HQL查询引擎及压缩接口将分布式Map压缩应用到连接查询的混洗阶段中,减小传递到查询聚合端的数据量,提高压缩数据的查询速度,并推导了时效性的相关公式。以北京某动车段10 kV电力远动监控系统的实测数据为例,搭建了四节点测试集群。压缩导入对比测试表明,分布式Map压缩速度快于分布式Reduce压缩,分布式Map的Map_Deflate压缩处理时间比分布式Reduce_Deflate减少了45.3%;压缩-查询测试表明,当数据量为2×10~7记录级时,分布式Map的Map_LZO格式压缩-查询耗时大幅降低,比混洗阶段不压缩-查询时减少了31.6%,验证了分布式Map压缩对加速查询的时效性。
[Abstract]:In order to deal with the problem of large amount of power monitoring data in distribution automation, a new distributed Map compression and query method for distribution network monitoring data is proposed. The distributed Map compression storage of monitoring data and the application of HQL query engine and compression interface to the mixed cleaning phase of the connection query are applied to reduce the distributed Map compression. It is transmitted to the amount of data at the aggregate end of the query, improves the query speed of the compressed data, and derives the relevant formulae of timeliness. Taking the measured data of the 10 kV electric power telecontrol monitoring system in a certain train section in Beijing as an example, the four node test cluster is set up. The compression import contrast test shows that the distributed Map compression speed is faster than the distributed Reduce compression. The Map_Deflate compression processing time of the distributed Map is reduced by 45.3% than that of distributed Reduce_Deflate; compression query testing shows that when the data amount is 2 x 10~7 record level, the Map_LZO format compression of distributed Map is greatly reduced, less than 31.6% when the query is not compressed to the query, which validates the distributed Map compression for accelerated query. It is timeliness.
【作者单位】： 华东交通大学电气与自动化工程学院;
【基金】：国家自然科学基金资助项目(51567008) 江西省杰出青年人才计划项目(20162BCB23045) 江西省自然科学基金资助项目(20161BAB206156,20171BAB206044)~~
【分类号】：TM76
，

本文编号：2019801