基于Hadoop的地质云计算平台搭建与应用

发布时间：2018-08-24 19:36

【摘要】：地质数据采集方式的多样性导致了数据规模的不断增长,已经达到了“地质大数据”的5“V”特点,数据管理和分析处理的复杂程度不断增加,使得对海量地质数据进行高效运维和数据挖掘的难度不断增大,迫切需要新的技术手段来实现地质数据的智能化服务和挖掘地质数据的潜在价值。分布式存储和云计算提供了一种解决上述问题的新思路。Hadoop大数据处理技术得到了国内外研究者们越多越多的关注,成为海量数据存储、计算、挖掘技术的研究热点。本文旨在基于搭建虚拟化地质云平台,实现积累的地质数据能够共享和互操作。深入研究和探索Hadoop集群中的HDFS分布式文件系统、Map Reduce并行编程框架、Hbase列式存储数据库等组件,结合全国地质矿产潜力评价数据,将Hadoop技术应用于地质大数据分析研究中。本文的主要工作如下:(1)通过对云计算和大数据的研究,阐述了其概念、关键技术等内容并提出了地质云平台的体系结构,重点阐述了开源云计算和存储框架Hadoop,尤其是分布式文件系统HDFS、并行计算框架Map Reduce和列式存储Hbase。(2)通过对海量地质数据整合、共享和查询检索的需求分析,利用分布式存储技术和虚拟化技术设计、搭建了Master/Slave架构的云数据计算与存储集群平台。利用Hadoop系统中的HDFS和Map Reduce,为我们设计海量地质数据存储架构提供了有力的技术支撑,最终实现在高并发、高负载的集群环境中对地质数据进行高效访问。(3)从Hadoop集群的云存储出发,解决了小文件在HDFS里合并存储的优化,使用Map Reduce算法使合并过程效率更高。同时通过整体考虑各个负载因素,采用信息熵算法确定权重值,经过多轮负载均衡,提高系统应对高并发情况,优化文件读写,系统效率有了极大提高。(4)研究了架构在虚拟云平台上的HBase数据库,根据矿产潜力评价数据的表特点设计rowkey,提高了地质大数据存储管理、查询检索的效率。通过与O racle关系数据库的数据入库、数据检索对比实验,验证了HBase在处理海量地质数据方面的优越性。
[Abstract]:The diversity of geological data collection methods has led to the continuous growth of data scale, which has reached the 5 "V" characteristic of "geological big data", and the complexity of data management and analysis has been increasing. It is becoming more and more difficult to carry out efficient transportation and data mining for massive geological data. Therefore, new technical means are urgently needed to realize the intelligent service of geological data and the potential value of mining geological data. Distributed storage and cloud computing provide a new way to solve the above problems. Hadoop big data technology has attracted more and more attention from researchers at home and abroad, and has become a research hotspot in mass data storage, computing and mining technology. The purpose of this paper is to share and interoperate the accumulated geological data based on the virtual geological cloud platform. In this paper, we deeply study and explore the components of HDFS distributed file system (HDFS) in Hadoop cluster, such as HDFS Reduce parallel programming framework, Hbase column storage database and so on. Combined with the evaluation data of geological and mineral potential in China, Hadoop technology is applied to the analysis and research of geological big data. The main work of this paper is as follows: (1) through the research of cloud computing and big data, the concept and key technology of cloud computing are expounded, and the architecture of geological cloud platform is put forward. The requirements of open source cloud computing and storage framework (Hadoop,), especially distributed file system (HDFS,) parallel computing framework (Map Reduce) and column storage Hbase. (2), are analyzed by integrating, sharing and querying massive geological data. A cloud data computing and storage cluster platform based on Master/Slave architecture is built by using distributed storage technology and virtualization technology. The use of HDFS and Map Reduce, in the Hadoop system provides a powerful technical support for us to design a massive geological data storage architecture, which is finally implemented in high concurrency. The geological data is accessed efficiently in the high-load cluster environment. (3) based on the cloud storage of Hadoop cluster, the optimization of merging and storing small files in HDFS is solved, and the Map Reduce algorithm is used to make the merging process more efficient. At the same time, considering all the load factors as a whole, using the information entropy algorithm to determine the weight value, after the multi-wheel load balancing, improve the system to deal with the high concurrency, optimize the file reading and writing, The system efficiency has been greatly improved. (4) the HBase database based on virtual cloud platform is studied. According to the table characteristics of mineral potential evaluation data, rowkey, is designed to improve the efficiency of geological big data storage management and query retrieval. The superiority of HBase in dealing with massive geological data is verified by the data input and data retrieval contrast experiment with O racle relational database.
【学位授予单位】：湖南科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：P628

【参考文献】