基于云计算的并行FFT算法及其在高铁数据中的应用研究

发布时间：2018-05-27 18:30

本文选题：高速铁路 + 云计算　；参考：《西南交通大学》2013年硕士论文

【摘要】：科技的不断进步推动了高速铁路的快速发展.而其作为一个节能、环保、快速和准点的交通工具日渐成为了人们出行的首选。随着高铁装备水平的不断改进和提高,安全性和舒适性也成为人们关注的焦点。为了保证列车的安全运行,我们需要实时地对列车运行数据进行分析处理,掌握列车的运营状态,做出决策判断。为了能够获得足够的高铁列车运行信息,工程人员利用大量的传感器来采集噪声和振动等多种类型的信号数据。然而,分析处理这种大规模增长的数据给传统的信号分析方法带来了前所未有的严峻挑战,而云计算作为一个新兴的并行处理技术,融合了网格计算、分布式计算和并行计算等特点,在大数据计算和网络存储方面具有卓越的性能表现。 Hadoop作为一种云计算框架,包含了文件系统HDFS和MapReduce编程模型,其具有高可靠性和高容错性等特点,尤其是MapReduce模型,采用分而治之的设计思想,有效地化解了大数据对现代程序设计过程带来的挑战。本文利用云计算技术对高铁数据处理领域十分重要的数据预处理和信号分析算法进行并行化,包括高铁原始数据解包算法和数字信号分析中广为应用的快速傅里叶变换算法FFT。高铁原始数据解包作为高铁数据预处理的第一步,为后期的数据预处理过程如数据平滑、去除异常点和去除线性趋势项等奠定了数据基础,对其并行化解决了传统解包算法处理测试数据集的瓶颈。实验证实,该算法在并行性方而表现良好。为了给工程人员提供一个系统化的数据预处理环境,本文设计了一个基于云计算的高铁数据预处理系统,其将多种并行数据预处理算法整合到一起,并且提供了对Hadoop集群的配置功能,工程人员只需要在系统中按需求提交处理任务,系统通过分析将任务操作步骤转交给Hadoop系统,待处理完毕之后,工程人员便可将处理结果下载到本地,极大地方便了数据预处理过程。 FFT作为离散傅里叶变换的一种快速算法,成为了数字信号分析领域中重要的工具,广泛应用于图像处理和通信技术等领域。高铁信号数据处理中也同样需要用到FFT算法,然而传统的串行FFT算法并不能适应大规模的高铁运行数据。于是,本文基于云计算技术设计了一个并行FFT算法,实验证明,该算法在准确率方而与串行算法结果保持一致,且节点间的并行性提升了运算效率,可以适应大规模的高铁数据集处理需求。
[Abstract]:The continuous progress of science and technology has promoted the rapid development of high-speed railway. As an energy-saving, environmental-friendly, fast and punctual transportation, it has become the first choice for people to travel. With the continuous improvement and improvement of high-speed equipment, safety and comfort have become the focus of attention. In order to ensure the safe operation of the train, we need to analyze and process the train operation data in real time, master the operation state of the train, and make the decision judgment. In order to obtain enough information of high-speed train operation, engineers use a large number of sensors to collect various kinds of signal data, such as noise and vibration. However, analyzing and processing this kind of large-scale growth data brings unprecedented challenges to traditional signal analysis methods. Cloud computing, as a new parallel processing technology, integrates grid computing. Distributed computing and parallel computing have excellent performance in big data computing and network storage. As a cloud computing framework, Hadoop includes file system HDFS and MapReduce programming model. It has the characteristics of high reliability and high fault tolerance, especially the MapReduce model, which adopts the design idea of divide-and-conquer. It effectively resolves the challenge brought by big data to the modern programming process. This paper uses cloud computing technology to parallelize the data preprocessing and signal analysis algorithms which are very important in the field of high-speed rail data processing, including the fast Fourier transform algorithm (FFTFT), which is widely used in high-speed rail raw data unpacking algorithm and digital signal analysis. As the first step of high-speed railway data preprocessing, the unpacking of raw data of high speed rail lays the data foundation for the later data preprocessing process such as data smoothing, removing abnormal points and removing linear trend items, etc. Parallelization solves the bottleneck of traditional unpacking algorithm in processing test data sets. Experiments show that the algorithm performs well in parallelism. In order to provide a systematic data preprocessing environment for engineers, this paper designs a high-speed railway data preprocessing system based on cloud computing, which integrates a variety of parallel data preprocessing algorithms. And the configuration function of Hadoop cluster is provided. Engineers only need to submit the processing task according to the requirement in the system. The system passes the task operation steps to the Hadoop system through the analysis, and after the processing is finished, Engineers can download the processing results to the local, which greatly facilitates the data preprocessing process. As a fast algorithm of discrete Fourier transform (DFT), FFT has become an important tool in the field of digital signal analysis, and has been widely used in image processing and communication technology. The FFT algorithm is also used in high-speed rail signal data processing, but the traditional serial FFT algorithm can not adapt to the large-scale high-speed rail operation data. Therefore, this paper designs a parallel FFT algorithm based on cloud computing technology. Experimental results show that the algorithm is consistent with the result of serial algorithm in accuracy, and the parallelism between nodes improves the computing efficiency. It can meet the needs of large-scale high-speed railway data set processing.
【学位授予单位】：西南交通大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP338.6

【参考文献】