基于Hadoop平台的风力发电机组振动数据存储技术研究
发布时间：2018-08-06 17:12
【摘要】：风力发电作为发展最快的新型清洁能源带动了风电技术的发展和广泛应用并促使风电机组的规模不断扩大,由此产生的数据规模也随之扩大。在风电机组处于工作状态时,其中的部件如齿轮箱、轴承等出现松动、磨损、异常等都会产生大量的振动数据,难以满足对海量数据整理、分析、存储需求,而且由于风电机的发电系统和监控设备的多样性,各类设备都产生不同的数据格式或数据类型,大都以数据流的形式输出。因此诸如Hadoop等云计算平台提供了对海量高维数据分析和处理的方式,为消耗大量资源的数据处理提供实时可靠相对廉价的计算资源。本文引入基于MapReduce的并行FFT算法和LZO压缩相结合的技术,对数据进行处理以降低网络传输量、减少存储所用空间。快速傅里叶变换FFT实现了信号从复杂的时域转换到具备显著特征的频域上,与离散傅里叶变换DFT相比大大减少了运算量,在数字系统、计算机系统等信号处理方面FFT的广泛应用是一个重大进展。FFT算法将数据以频域形式展现,能够有效的分析数据的特征、设备状态、进行故障诊断等。本文研究了FFT算法的原理和特点,采用在MapReduce的模型上实现了FFT的并行化。根据FFT的特点将算法的并行执行分为数据补齐、变址运算、蝶形运算、格式化四个阶段,为压缩提供可靠的基础,加强数据存储的效率。面对庞大数据量的存储,采用压缩技术不仅节省空间,还可以降低数据文件传输过程中的I/O,本文在FFT每一阶段的中间结果和最终结果中嵌入压缩技术,保证了数据压缩的有效性。首先引入了Hadoop支持的Gzip、Bzip2、LZO三种压缩格式,对三种压缩格式的压缩率和压缩速度进行测试比较得出了适用本文的压缩格式,随后着重研究了该压缩方式的实现过程、分析压缩性能,实现对振动数据的压缩存储。
[Abstract]:Wind power, as the fastest developing new clean energy, promotes the development and wide application of wind power technology, and makes the scale of wind turbine expand constantly, and the resulting data scale also expands. When the wind turbine is in the working state, the components such as gearbox, bearing and so on will be loosened, worn, abnormal and so on will produce a large amount of vibration data, it is difficult to meet the demand of sorting, analyzing and storing the massive data. Because of the diversity of generating system and monitoring equipment of wind motor, all kinds of equipments produce different data format or data type, most of them are outputted in the form of data stream. Therefore cloud computing platforms such as Hadoop provide a way to analyze and process massive high-dimensional data and provide real-time reliable and relatively cheap computing resources for data processing which consumes a lot of resources. In this paper, the parallel FFT algorithm based on MapReduce and the technology of LZO compression are introduced to process the data in order to reduce the amount of network transmission and reduce the space used for storage. Fast Fourier transform (FFT) realizes signal conversion from complex time domain to frequency domain with obvious characteristics. Compared with discrete Fourier transform (DFT), it greatly reduces the computation. The wide application of FFT in signal processing, such as computer system, is a great progress. The algorithm presents the data in frequency domain, which can effectively analyze the characteristics of data, equipment status, fault diagnosis and so on. In this paper, the principle and characteristics of FFT algorithm are studied, and the parallelization of FFT based on MapReduce model is implemented. According to the characteristics of FFT, the parallel execution of the algorithm is divided into four stages: data completion, indexing, butterfly operation and formatting, which provide a reliable basis for compression and enhance the efficiency of data storage. In the face of the huge amount of data storage, the compression technology not only saves space, but also reduces the I / O ratio in the process of data file transfer. This paper embed compression technology in the intermediate and final results of each stage of FFT. The validity of data compression is ensured. Firstly, three compression schemes supported by Hadoop are introduced. The compression ratio and compression speed of the three compression formats are tested and compared, and the compression format suitable for this paper is obtained. Then, the realization process of the compression scheme is studied emphatically, and the compression performance is analyzed. The compression storage of vibration data is realized.
【学位授予单位】：华北电力大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TM614
[Abstract]:Wind power, as the fastest developing new clean energy, promotes the development and wide application of wind power technology, and makes the scale of wind turbine expand constantly, and the resulting data scale also expands. When the wind turbine is in the working state, the components such as gearbox, bearing and so on will be loosened, worn, abnormal and so on will produce a large amount of vibration data, it is difficult to meet the demand of sorting, analyzing and storing the massive data. Because of the diversity of generating system and monitoring equipment of wind motor, all kinds of equipments produce different data format or data type, most of them are outputted in the form of data stream. Therefore cloud computing platforms such as Hadoop provide a way to analyze and process massive high-dimensional data and provide real-time reliable and relatively cheap computing resources for data processing which consumes a lot of resources. In this paper, the parallel FFT algorithm based on MapReduce and the technology of LZO compression are introduced to process the data in order to reduce the amount of network transmission and reduce the space used for storage. Fast Fourier transform (FFT) realizes signal conversion from complex time domain to frequency domain with obvious characteristics. Compared with discrete Fourier transform (DFT), it greatly reduces the computation. The wide application of FFT in signal processing, such as computer system, is a great progress. The algorithm presents the data in frequency domain, which can effectively analyze the characteristics of data, equipment status, fault diagnosis and so on. In this paper, the principle and characteristics of FFT algorithm are studied, and the parallelization of FFT based on MapReduce model is implemented. According to the characteristics of FFT, the parallel execution of the algorithm is divided into four stages: data completion, indexing, butterfly operation and formatting, which provide a reliable basis for compression and enhance the efficiency of data storage. In the face of the huge amount of data storage, the compression technology not only saves space, but also reduces the I / O ratio in the process of data file transfer. This paper embed compression technology in the intermediate and final results of each stage of FFT. The validity of data compression is ensured. Firstly, three compression schemes supported by Hadoop are introduced. The compression ratio and compression speed of the three compression formats are tested and compared, and the compression format suitable for this paper is obtained. Then, the realization process of the compression scheme is studied emphatically, and the compression performance is analyzed. The compression storage of vibration data is realized.
【学位授予单位】：华北电力大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TM614
【参考文献】
相关期刊论文 前10条
1 杜江;张铮;张杰鑫;邰铭;;MapReduce并行编程模型研究综述[J];计算机科学;2015年S1期
2 向丽辉;缪力;张大方;;压缩对Hadoop性能影响研究[J];计算机工程与科学;2015年02期
3 孙知信;黄涵霞;;基于云计算的数据存储技术研究[J];南京邮电大学学报(自然科学版);2014年04期
4 罗鹏;龚勋;;HDFS数据存放策略的研究与改进[J];计算机工程与设计;2014年04期
5 李伟卫;赵航;张阳;王勇;;基于MapReduce的海量数据挖掘技术研究[J];计算机工程与应用;2013年20期
6 翟岩龙;罗壮;杨凯;徐晟晨;;基于Hadoop的高性能海量数据处理平台研究[J];计算机科学;2013年03期
7 郝树魁;;Hadoop HDFS和MapReduce架构浅析[J];邮电设计技术;2012年07期
8 周超;朱q，
本文编号：2168401
本文链接：https://www.wllwen.com/kejilunwen/dianlidianqilunwen/2168401.html
