大数据技术在电力营销系统中的应用研究

发布时间：2018-04-23 04:42

本文选题：Spark + FCM聚类算法　；参考：《华北电力大学》2017年硕士论文

【摘要】：随着电力企业的快速发展,电网数据不断积累,呈现出爆炸式增长的趋势。怎样从这些数据中挖掘出价值数据,已经成为智能电网建设和电力企业发展的关键,而大数据相关技术的飞速发展为电网数据带来新机遇。在当下主流的大数据技术中,相比Hadoop平台,Spark平台更加优异的迭代工作负载性能、RDD数据集的高速计算能力以及HDFS的强大存储能力在海量的电力数据挖掘中都表现出极大的优势。本文利用Spark平台与关联规则挖掘技术结合的优势,根据电力企业对数据分析的实际需求,研究分析了Z-score标准化和FCM聚类算法,结合两者设计出电力数据预处理流程,验证了数据预处理的有效性,并运用稀疏矩阵和FCM聚类算法改进了FFP-growth算法,应用于Spark平台下的电力营销系统中。本文主要完成了以下三方面工作:第一,研究实际处理电网数据中遇到的问题,并针对这些问题提出解决方法,根据Z-score标准化和FCM聚类算法的结合提出大数据预处理的新流程;第二,研究了FP-growth关联规则挖掘算法,根据传统FP-Growth算法的不足对其进行改进,引入稀疏矩阵存储减少一次数据库扫描得到频繁项集,节省了数据库扫描时间,并设计了在Spark平台下改进的FFP-growth算法并行化模型,对数据进行二次分类,避免单个事务集过大的问题;第三,搭建Spark平台,设计并实现电力营销系统,将数据预处理技术和改进的FFP-Growth关联规则挖掘技术应用到电力营销数据分析系统中,实例验证了重新设计的数据预处理流程和改进的并行关联规则挖掘算法在电力营销数据分析中的有效性。
[Abstract]:With the rapid development of power enterprises, power grid data accumulation, showing an explosive growth trend. How to excavate the value data from these data has become the key to the construction of smart grid and the development of electric power enterprises, and the rapid development of big data technology brings new opportunities for power grid data. In the current mainstream big data technology, the iterative workload performance of Hadoop platform is better than that of Hadoop platform. The high speed computing power of RDD data set and the powerful storage capacity of HDFS all show great advantages in massive power data mining. This paper makes use of the advantage of combining Spark platform with association rule mining technology, according to the actual demand of data analysis in electric power enterprises, studies and analyzes the Z-score standardization and FCM clustering algorithm, and designs the power data preprocessing flow combining the two technologies. The validity of data preprocessing is verified, and the FFP-growth algorithm is improved by using sparse matrix and FCM clustering algorithm, which is applied to the electric power marketing system based on Spark platform. This paper mainly completes the following three aspects of work: first, study the problems encountered in the actual processing of power grid data, and propose solutions to these problems, according to the combination of Z-score standardization and FCM clustering algorithm proposed big data pre-processing new flow; Secondly, the FP-growth association rule mining algorithm is studied, which is improved according to the shortcomings of the traditional FP-Growth algorithm. The sparse matrix storage is introduced to reduce the frequent itemsets of database scanning, which saves the scanning time of the database. And designed an improved FFP-growth algorithm parallelization model under the Spark platform, the data is classified twice to avoid the problem of a single transaction set too large. Thirdly, build the Spark platform, design and implement the electric power marketing system. The data preprocessing technology and the improved FFP-Growth association rule mining technology are applied to the power marketing data analysis system. Examples show that the redesigned data preprocessing process and the improved parallel association rule mining algorithm are effective in power marketing data analysis.
【学位授予单位】：华北电力大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：F426.61;TP311.13

【参考文献】