分布式文件系统中的若干关键算法研究

发布时间：2018-04-12 16:17

本文选题：锁无关消息队列 + 流水线　；参考：《南京大学》2013年硕士论文

【摘要】：随着社会、经济、互联网的高速发展,全球每年产生的数据量正呈几何数量级的速度增长,人类已经进入了大数据的时代。如何高效、可靠的存储、访问这些大数据,一直是企业界、学术界研究的热点。过去的数据存储大都是集中式的存储方式,而随着数据量的高速增长,集中式的存储方式在面对日益增长的存储、访问需求时,计算资源、IO资源、网络资源和存储资源都将会出现瓶颈。近十年来,人们意识到,可以利用大量廉价的计算机组成的计算机集群来解决日益增长的存储、计算、网络访问需求。现在,集群已经成为主流的高性能计算机体系架构。在集群上搭建分布式文件系统,也成为当今解决海量数据存储、访问的主流解决方案。本文针对面向集群计算的分布式文件系统中的若干关键问题进行了研究,主要工作如下：◇提出并实现了一种锁无关的消息队列访问算法。当前市场上流行的分布式文件系统中的每个服务器大都采取了多线程的架构方案。网络模块是一个单独的线程,通过一个基于锁的消息队列和其它处理业务逻辑的线程进行通信。当分布式文件系统中访问量增大时,锁冲突造成的效率下降是网络模块的一个性能瓶颈。本文提出了一种锁无关的网络消息队列访问算法,有效地解决了一个读者、一个写者环境下锁带来的性能问题。◇提出并实现了一种支持流水线式访问的多数据服务器访问算法。分布式文件系统中的数据是分散存储在不同数据服务器上的,同时访问多个数据需要一个分布式互斥算法,当前的分布式文件系统大多采取锁管理器的方法。本文提出了一种新的分布式互斥算法,充分利用了分布式集群中数据访问的“流水线”特征,使得资源利用最大化,有效解决了锁管理服务器带来的访问延迟问题。◇提出并实现了一种支持动态增减决策者的分布式一致性算法。分布式一致性算法是分布式文件系统中使用的基础算法之一,传统的分布式一致性算法,例如Paxos算法,大都不支持增减决策者。本文提出了改进的Paxos算法,可以在决策的过程中动态增减决策者,有效解决了使用传统一致性算法时无法扩展系统的问题。
[Abstract]:With the rapid development of society, economy and Internet, the amount of data produced in the world is increasing at a geometric order of magnitude every year. Human beings have entered the era of big data.How to store and access these big data efficiently and reliably has always been a hot spot in business and academic circles.In the past, most of the data storage was centralized storage, but with the rapid growth of data, the centralized storage in the face of increasing storage, access requirements, computing resources IO resources,Network resources and storage resources will appear bottleneck.In the last decade, people have realized that a large number of cheap computer clusters can be used to solve the growing demand for storage, computing, and network access.Now, cluster has become the mainstream of high-performance computer architecture.Building distributed file system on cluster has become the mainstream solution for mass data storage and access.In this paper, some key problems in distributed file system for cluster computing are studied. The main work is as follows: a lock independent message queue access algorithm is proposed and implemented.At present, most of the distributed file systems in the market adopt multi-thread architecture.A network module is a single thread that communicates through a lock-based message queue and other threads that handle business logic.When the traffic in distributed file system increases, the efficiency due to lock conflict is a bottleneck of network module performance.In this paper, a locking independent network message queue access algorithm is proposed, which effectively solves the problem of a reader.This paper proposes and implements a multi-data server access algorithm which supports pipeline type access.The data in distributed file system is distributed and stored on different data servers. It needs a distributed mutex algorithm to access multiple data at the same time. Most of the current distributed file systems adopt the method of lock manager.In this paper, a new distributed mutex algorithm is proposed, which makes full use of the "pipeline" feature of data access in distributed cluster, and makes the maximum utilization of resources.The problem of access delay caused by lock management server is solved effectively. A distributed consistency algorithm supporting dynamic decision makers is proposed and implemented.Distributed consistency algorithm is one of the basic algorithms used in distributed file system. Most of the traditional distributed consistency algorithms, such as Paxos algorithm, do not support the increase or decrease of decision makers.In this paper, an improved Paxos algorithm is proposed, which can dynamically increase or decrease the decision makers in the process of decision making, and effectively solves the problem that the system can not be extended by using the traditional consistency algorithm.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP338.8

【相似文献】