基于Hadoop的煤炭企业数据共享平台设计与实现

发布时间：2018-07-10 07:15

本文选题：信息孤岛 + 海量数据　；参考：《西安科技大学》2015年硕士论文

【摘要】：煤炭在我国能源消费中占有很大比重,而大型国有煤炭企业在我国煤炭生产领域掌握着统治地位,为保障我国的能源安全、经济发展和社会稳定做出了突出贡献。为了提高企业的生产效率、降低企业运营成本、预防煤矿安全事故,煤炭企业纷纷建立了自己的信息化系统。但由于在建设时没有进行合理的规划,不同信息化系统之间数据不能共享,在企业内部形成众多“信息孤岛”。一些煤炭企业为了打破“信息孤岛”,陆续建立起了数据共享平台,但这些数据共享平台无法满足当前煤炭企业信息化系统中的海量数据处理需求。Hadoop是一个分布式系统基础架构,依靠集群的方式在廉价机器上提供高速运算和海量存储。Hadoop使运行和处理大数据更加方便,为解决煤炭企业数据中心存在的问题提供了一条有效的途径。论文通过对于煤炭企业数据中心所面临的大数据处理难题的研究和分析,首先建立了煤炭企业数据共享模型,该模型在Hadoop上建立一个数据仓库,从源数据库按照要求抽取的数据经过数据集成后存储于数据仓库中,通过向外提供统一的数据接口,为用户提供数据访问和数据分析的服务。其次,针对某煤炭企业的实际需求,以该煤炭企业的生产数据为例,完成了数据共享平台的设计。完成了Hadoop平台和服务器端数据模型的设计。对应用Sqoop进行数据抽取的过程进行了阐述。定义了数据清洗和转换的要求和方法。详细介绍了应用的并行FP-Growth算法的算法流程,并设计了几个简单示例应用。最后,对平台功能进行了力所能及的实现。依据设计要求,通过Sqoop将数据从源数据库中抽取出来,放入Hive数据仓库中。Hive和Eclipse集成,在Eclipse下编程实现数据的清洗、转换和加载。应用Eclipse调用Mahout的并行FP-Growth方法进行数据分析,并将结果保存在数据库中。将数据处理结果导入到平台数据库中,并基于Spring架构,在服务器端完成了示例应用的开发。基于Hadoop的煤炭企业数据共享平台,利用Hadoop平台采用成熟的开源的技术,拥有强大的处理能力和高稳定性,不仅提高了实现数据共享的效率,同时可以满足对这些数据进行分析和处理的需求,而且降低了企业数据中心的运营成本。该平台有效的解决了煤炭企业信息系统中普遍存在的“信息孤岛”和海量数据的问题,煤炭企业可以基于此平台进一步开发特定的功能应用,为企业生产经营提供帮助。
[Abstract]:Coal occupies a large proportion in China's energy consumption, while large state-owned coal enterprises hold a dominant position in the field of coal production in China, which has made outstanding contributions to ensuring the energy security, economic development and social stability of our country. In order to improve the production efficiency, reduce the operation cost and prevent coal mine safety accidents, coal enterprises have established their own information systems. However, due to the lack of reasonable planning in construction, the data between different information systems can not be shared, forming a large number of "information isolated islands" in the enterprise. In order to break the "information island", some coal enterprises have set up a data-sharing platform one after another. However, these data sharing platforms can not meet the needs of mass data processing in the current information system of coal enterprises. Hadoop is a distributed system infrastructure. Relying on cluster to provide high speed operation and mass storage. Hadoop makes it more convenient to run and process big data on cheap machines. It provides an effective way to solve the problems existing in data centers of coal enterprises. Based on the research and analysis of the big data processing problem faced by the coal enterprise data center, this paper first establishes the coal enterprise data sharing model, and the model establishes a data warehouse on Hadoop. The data extracted from the source database is stored in the data warehouse after data integration. By providing a unified data interface, the data access and data analysis services are provided for the users. Secondly, according to the actual demand of a coal enterprise, taking the production data of the coal enterprise as an example, the design of the data sharing platform is completed. The design of Hadoop platform and server-side data model is completed. The process of data extraction using Sqoop is described. The requirements and methods of data cleaning and conversion are defined. The algorithm flow of parallel FP-Growth algorithm is introduced in detail, and several simple examples are designed. Finally, the platform function can be realized. According to the design requirements, the data is extracted from the source database through Sqoop, and put into the Hive data warehouse. Hive integrates with Eclipse, and the data is cleaned, transformed and loaded by programming under Eclipse. Using Eclipse to call Mahout's parallel FP-Growth method to analyze the data and save the results in the database. The data processing result is imported into the platform database, and based on Spring architecture, the example application is developed on the server side. The coal enterprise data sharing platform based on Hadoop, using Hadoop platform with mature open source technology, has strong processing ability and high stability, which not only improves the efficiency of data sharing, but also improves the efficiency of data sharing. At the same time, it can meet the needs of data analysis and processing, and reduce the operating cost of enterprise data center. The platform effectively solves the problems of "information isolated island" and massive data in the information system of coal enterprises. Coal enterprises can further develop specific functional applications based on this platform to provide help for production and management of enterprises.
【学位授予单位】：西安科技大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：F426.21;TP311.13

【相似文献】