大规模键值分布式存储系统的设计与实现

发布时间：2018-04-21 21:45

本文选题：分布式存储系统 + 散列　；参考：《电子科技大学》2013年硕士论文

【摘要】：随着互联网的不断发展，全球网络化的不断普及，海量信息的不断聚合，数据量保持指数级增长。面对当前的海量数据，传统的单机文件存储系统并不能提供其所需要的存储能力和高性能读、写需求，同时，传统文件系统的扩展性不佳也使得当数据业务需求发生变化时服务能力不足。鉴于此，本文中提出了一种新型的分布式键值存储系统的架构模型，该模型通过结合散列算法和一致性哈希算法，不仅解决了已有分布式系统中存在的扩展性问题和节点负载均衡问题，同时有效的避免了当机器数目发生改变时，数据移动代价过高的问题。本文的主要工作有：第一，介绍了当前管理海量数据的主要问题，以及本文的研究动机。第二，详细分析了已有的分布式存储系统，包括NFS系统，HDFS系统，DYNAMO系统等当前比较主流的分布式文件存储系统，这些系统的相关核心技术对于本文中设计的分布式存储系统具有良好的借鉴性和参考性。第三，在相关技术准备完成后，本文设计了一种新的大规模键值分布式存储系统的架构方式，该系统使用三层的架构模式，同时在三层中都设计了良好的灾备方案，从前端服务器层，到最底层的数据节点层，都尽可能的在保证高效性的前提下设计了良好的灾备机制，虽然数据备份牺牲了一定的数据一致性，但仍可以很好的满足应用需求。第四，完成系统设计后，本文中使用python语言对系统进行了仿真，将三层架构分解为三个实现模块，并将每一个模块再次详细拆分为不同功能的子模块，使得整个系统的耦合性降到最低。第五，为了保证系统的功能完备性和可靠性，本文对该系统做了细致的测试，包括功能测试和性能测试，测试结果完全符合预期的设计目标。第六，在论文的结束部分分析了一些可以优化本系统的技术和实现方式，并展望了分布式存储系统良好的发展前景。本文中成功的设计并实现了一个新型的大规模分布式存储系统，，对分布式存储系统的发展具有良好的探索和实践意义。
[Abstract]:With the development of the Internet, the popularization of the global network and the continuous aggregation of massive information, the amount of data keeps increasing exponentially. In the face of the current massive data, the traditional single-machine file storage system can not provide the required storage capacity and high performance read and write requirements, at the same time, The poor scalability of traditional file systems also leads to inadequate service capacity when data business requirements change. In view of this, a new architecture model of distributed key-value storage system is proposed in this paper, which combines hash algorithm and consistent hash algorithm. It not only solves the problem of scalability and node load balancing in existing distributed systems, but also effectively avoids the problem of excessive cost of data movement when the number of machines changes. The main work of this paper is as follows: First, it introduces the main problems of managing massive data and the motivation of this paper. Secondly, the existing distributed storage system, including NFS system and DYNAMO system, are analyzed in detail. The core technologies of these systems have good reference and reference for the distributed storage system designed in this paper. Third, after the completion of the related technology preparation, this paper designs a new large-scale key-value distributed storage system architecture, the system uses a three-tier architecture model, at the same time in the three-tier design of a good disaster preparedness scheme. From the front-end server layer to the bottom layer of the data node layer, we have designed a good disaster preparedness mechanism under the premise of high efficiency. Although the data backup sacrifices some data consistency, it can still meet the application requirements very well. Fourth, after completing the system design, this paper uses python language to simulate the system, decomposing the three-tier architecture into three implementation modules, and dividing each module into sub-modules with different functions in detail. The coupling of the whole system is minimized. Fifthly, in order to ensure the functional completeness and reliability of the system, the system is tested in detail, including function test and performance test, and the test results are in line with the expected design objectives. Sixth, at the end of the thesis, we analyze some technologies and implementation methods that can optimize the system, and look forward to the future of distributed storage system. In this paper, a new large-scale distributed storage system is successfully designed and implemented, which has a good exploration and practical significance for the development of distributed storage system.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】