基于Hadoop的煤矿设备数据规范化和清洗的研究

发布时间：2018-09-01 08:21

【摘要】：随着网络信息化技术快速发展和煤矿机械设备日趋大型化、复杂化,煤矿设备的运行数据规模不断地扩展,数据量从GB级迈向TB级甚至PB级,而且数据的差异性也越来越大。对于这种煤矿设备的海量异构数据,传统的数据分析处理系统已经不能满足数据处理和深度挖掘的需求,云计算技术可以高效地实现对这些问题处理。针对现有煤矿机械设备系统的实际情况和需求,分析云计算技术在工业化生产中广泛有效的应用,建立了基于Hadoop平台的煤矿设备数据处理系统,利用平台的海量存储HDFS和分布式计算Mapreduce技术,实现对这些海量级数据的处理、分析和深度挖掘。首先,针对煤矿设备海量异构格式的数据需要规范化描述问题,设计了用XML数据格式做为多样化海量数据的一种应用标准,分析元数据抽取和关系—映射模型技术,实现对煤矿设备异构数据的规范化描述。其次,在实验室环境下搭建了Linux集群和Hadoop集群的平台,建立基于Hadoop的煤矿设备系统数据处理系统,并对平台所用到HDFS技术和分布式计算Mapreduce技术进行了详细分析,实现对煤矿设备数据的海量存储和高效计算。最后,在Hadoop平台上建立对煤矿设备振动数据进行特征值提取和异常点清洗的整体架构,并提出了相应的Mapreduce算法,可以有效地实现对煤矿机械设备振动数据的处理。通过利用该系统平台对设备数据进行HDFS的读写和Mapreduce处理等性能测试,结果表明,Hadoop平台系统能够正常高效运行。因此,基于Hadoop的煤矿设备数据处理系统能够为煤矿设备维护提供很好的技术支持。
[Abstract]:With the rapid development of network information technology and the increasingly large-scale and complex of coal mine machinery and equipment, the operation data scale of coal mine equipment is constantly expanding, and the data volume is moving from GB level to TB level or even PB level, and the difference of data is becoming larger and larger. For the massive heterogeneous data of this kind of coal mine equipment, the traditional data analysis and processing system can no longer meet the demand of data processing and deep mining. Cloud computing technology can effectively deal with these problems. According to the actual situation and demand of the existing coal mine mechanical equipment system, this paper analyzes the wide and effective application of cloud computing technology in industrial production, and establishes a coal mine equipment data processing system based on Hadoop platform. The massive storage HDFS and distributed computing Mapreduce technology of the platform are used to realize the processing, analysis and deep mining of these magnanimous data. First of all, aiming at the problem that the data of massive heterogeneous format of coal mine equipment needs to be standardized, this paper designs an application standard of using XML data format as a kind of diversified mass data, analyzes metadata extraction and relational mapping model technology. The standard description of heterogeneous data of coal mine equipment is realized. Secondly, the platform of Linux cluster and Hadoop cluster is built in the laboratory environment, and the data processing system of coal mine equipment system based on Hadoop is established, and the HDFS technology and distributed computing Mapreduce technology used in the platform are analyzed in detail. To realize the massive storage and efficient calculation of coal mine equipment data. Finally, the whole structure of extracting the eigenvalue and cleaning the abnormal points of coal mine equipment vibration data is established on Hadoop platform, and the corresponding Mapreduce algorithm is put forward, which can effectively process the vibration data of coal mine equipment. By using the system platform to test the performance of HDFS reading and writing and Mapreduce processing, the results show that the system can run normally and efficiently. Therefore, the coal mine equipment data processing system based on Hadoop can provide good technical support for coal mine equipment maintenance.
【学位授予单位】：西安科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TD407;TP311.13

【相似文献】