HDFS文件系统中元数据的高可用性管理方法研究

发布时间：2018-01-02 21:25

本文关键词：HDFS文件系统中元数据的高可用性管理方法研究　出处：《华南理工大学》2013年硕士论文　论文类型：学位论文

【摘要】：随着全球互联网的快速延伸和各行各业信息化技术的快速发展，各种数据信息量在本世纪开始以来的12年时间里已经呈指数级的增长趋势。传统的存储系统显然已无法在存储容量和数据存储管理方面满足海量数据存储的需求，而分布式存储系统恰好在存储容量和数据存储管理方面较好地弥补了传统存储技术的不足；因此，，分布式存储系统技术已经成为当前计算机信息存储技术领域内的“热点”研究方向。分布式文件系统在分布式存储系统中的作用非常重要，它是实现分布式存储的核心技术，所以对分布式文件系统进行研究有重要的现实意义。 HDFS文件系统是一个开源的分布式存储文件系统，目前在分布式存储系统中的应用比较多。但是由于HDFS文件系统一般是将元数据存放在存储系统中一个单独的节点上，这种元数据布局方式给分布式存储系统的I/O性能带来了不利影响，另外单一节点还存在单点隐患问题，因此很难保障元数据管理的可用性和稳定性。针对以上问题，本文重点对HDFS文件系统中的元数据的高可用性管理方法进行了比较深入的研究，论文作者所做的主要研究工作和取得的成果如下： 1、在分析和研究了HDFS元数据管理原理的基础上，提出了一种HDFS文件系统中元数据的高可用性管理方案。 2、给出了一种业务节点与元数据节点通信的改进方法。因为分布式存储的一个重要特点就是元数据与业务数据分开管理，而元数据节点的结构变化会引起业务数据节点的通信机制产生变化，所以改进业务节点与元数据节点之间的通信方式非常重要。 3、设计了一个基于HDFS的双节点元数据管理系统。该元数据管理系统包含了两个并列的元数据节点，每个节点可以同时处理来自客户端的请求，同时元数据节点之间可以实时进行数据的一致性通信；当某一节点失效时，存储系统不需要进行数据迁移仍然可以稳定地对外进行服务。本文已将改进后的HDFS文件系统应用到了作者单位的“综合平台用户行为分析引擎关键技术现场试验”项目中，并与改进前的HDFS进行了完整的对比测试。测试数据表明改进后的双节点元数据管理系统在稳定性方面效果比较明显，它提升了整个HDFS文件系统的可用性和稳定性。
[Abstract]:With the rapid development of the rapid extension of the global Internet and information technology in all walks of life, all kinds of data in 12 years since the beginning of this century there has been increasing exponentially. The traditional storage system obviously has not in the storage capacity and data storage management aspects to meet the massive data storage requirements, the distributed storage system just in the storage capacity and data storage management can make up the shortcomings of traditional storage technology; therefore, the distributed storage system technology has become the current computer information storage technology in the field of "hot spot" in the research direction.
Distributed file system plays a very important role in distributed storage system. It is the core technology to realize distributed storage. Therefore, the research of distributed file system has important practical significance.
The HDFS file system is a distributed storage file system is an open source, there are many applications in distributed storage system. But because the HDFS file system is stored in the metadata storage system in a single node, the metadata layout to bring a negative impact on the performance of I/O distributed storage system, in addition to a single node there is a single point problem, so it is difficult to guarantee the availability and stability of metadata management.
In view of the above problems, this paper focuses on the high availability management method of metadata in HDFS file system.
1, on the basis of analyzing and studying the principle of HDFS metadata management, a high availability management scheme for meta data in HDFS file system is proposed.
2, the improved method is a service node and metadata node communication. Because separate management is an important feature of distributed storage is the metadata and business data, and the communication mechanism of changes in the structure of metadata node will cause the business data node changes, so the improved means of communication between service nodes and metadata nodes is very important.
3, the design of a HDFS double point. The metadata management system based on metadata management system contains two parallel metadata nodes, each node can handle the request from the client, and the consistency of metadata node between the communication of real-time data; when a node fails, the system does not require data storage migration can still be stable in foreign service.
This paper has improved HDFS file system applied to the Department of "integrated platform user behavior analysis engine key technology field test project, and compared the complete test and improvement of the HDFS. The test data shows that two node metadata management system improved results in stability obviously, raise it the availability and stability of the HDFS file system.

【学位授予单位】：华南理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】