基于云存储的分布式文件系统研究与优化

发布时间：2018-05-30 21:09

本文选题：云存储 + HDFS　；参考：《西安电子科技大学》2013年硕士论文

【摘要】：随着互联网的飞速发展，，全球数据量每年以指数增长，使得云计算成为了当前研究与应用的热点。云存储作为云计算的底层服务，是一种架构复杂的分布式文件系统。因为它具有结构灵活、响应效率高、管理方便等优点，因而成为世界各国解决数据爆炸性增长方案的首选。Hadoop分布式文件系统（HDFS）作为当今最流行的基于云存储的分布式文件系统具有开源、廉价、高容错以及高可扩展性的特点，在云存储领域占居了者举足轻重的地位。然而，HDFS因其结构和性能上的局限性，也存单点失效、并发用户的高延时访问、负载均衡不足等的问题。本文在系统、全面的学习和总结分布式存储系统发展现状和特点的基础上，重点分析了几种常用的分布式存储系统架构的优缺点，同时设计了一个部分对等式的多Namenode系统架构。该架构通过增加元数据服务器层中部分对等的多个Namenode，改变了以HDFS为代表的集中式存储系统对主节点的单点依赖，降低了并发用户的等待时延和元数据服务器的平均内存占用率。同时，本文还深入研究了常用的负载均衡方法，针对HDFS存储服务器负载均衡不足的缺点，建立了磁盘利用率模型和服务阻塞率模型，设计了一种基于本文架构的自适应反馈负载均衡算法。通过算法性能分析与实验仿真进一步论证了本文设计的算法比HDFS系统中的负载均衡算法在系统性能和负载均匀度方面都有一定的优化。
[Abstract]:With the rapid development of the Internet, the global data volume increases exponentially every year, which makes cloud computing become the focus of current research and application. Cloud storage, as the underlying service of cloud computing, is a complicated distributed file system. Because it has the advantages of flexible structure, high response efficiency, convenient management and so on. Therefore, as the most popular distributed file system based on cloud storage, the Hadoop distributed file system (HDFS) is the most popular distributed file system based on cloud storage, which has the characteristics of open source, low cost, high fault tolerance and high scalability. Cloud storage occupies a pivotal position in the field of cloud storage. However, due to its limitations in structure and performance, HDFS also has some problems, such as failure of single point, high delay access of concurrent users, insufficient load balancing, and so on. On the basis of studying and summarizing the current situation and characteristics of distributed storage system, this paper analyzes the advantages and disadvantages of several commonly used distributed storage system architectures, and designs a partial peer-to-peer multi-Namenode system architecture. By adding several Namenodes in the metadata server layer, the architecture changes the single point dependence of the centralized storage system represented by HDFS on the master node, and reduces the waiting delay of concurrent users and the average memory occupancy of metadata server. At the same time, this paper also deeply studies the commonly used load balancing methods, aiming at the disadvantage of insufficient load balance of HDFS storage server, the disk utilization model and the service blocking rate model are established. An adaptive feedback load balancing algorithm based on this architecture is designed. Through the performance analysis and experimental simulation, it is further demonstrated that the proposed algorithm is better than the load balancing algorithm in HDFS system in terms of system performance and load uniformity.
【学位授予单位】：西安电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】