基于CEPH分布式文件系统的云盘系统的设计与实现
发布时间：2018-12-30 21:11
【摘要】：数据是当今互联网时代最核心最具价值的产物之一,个人在消费和使用互联网产品的同时产生了大量需要持久化存储的数据,个体存储设备的解决方式在这样一个存储量和存储要求日益增长的条件下不再能够满足个人用户的要求,因此越来越多的个人用户成为了云存储的受众。云计算几十年的发展使得公有云环境下的存储技术已然十分成熟,商业环境下诞生了许多优秀的云盘产品,但其核心技术是对外封闭的,而对于这种面向大量个人用户的公有云盘系统,开源社区也并没有完善可靠的整体解决方案。本文基于上述原因将公有云盘涉及的各个关键模块和技术点进行逐个研究和设计,提出了一整套可靠的云盘系统实现架构。首先对云盘的远程文件系统操作机制的原理和技术进行了分析,研究并设计出CFSRPC协议,给出了协议的客户端和服务端的具体格式定义和二者的通信机制,云盘系统通过对该协议的实现,可以透明化远程文件操作的细节,方便客户端和服务端上层业务的实现。与此同时为该协议设计了事务机制以颗粒化文件传输服务,设计了CSDA算法在协议级别动态调节文件传输的分块大小,增加了该协议对云盘系统性能的优化。接着对云盘系统的核心业务流程进行分析,针对云盘系统依赖的多个关键模块进行了分析和整合设计,总结出以访问层、管理层、存储层为核心的整体软件架构。基于Ceph分布式文件系统设计核心存储模块,研究分析CephFS和MDS的实现原理并根据云盘系统架构和CFSRPC协议的定义,结合多路IO复用机制设计实现了Ceph的接入模块CCD。将云盘系统的请求分化为文件系统操作请求和管理功能操作请求,分别以CCD模块和管理功能模块来接受和处理,中间加入LVS+Keepalived的负载机制使得CCD节点和管理功能节点支持横向扩展。参考Keystone的令牌认证机制设计实现了独立认证系统。以NW.js为框架根据CFSRPC协议的定义设计实现了客户端。最后将整套云盘系统在测试环境下进行部署,完成了认证流程和云盘基本操作的测试,并对单点CCD进行了性能测试,确认整体系统能够达到公有云生产环境的性能要求。
[Abstract]:Data is one of the core and most valuable products in the Internet age. While consuming and using Internet products, individuals have produced a large number of data that need to be persisted and stored at the same time. The solution of individual storage devices can no longer meet the requirements of individual users under the condition of increasing storage capacity and storage requirements, so more and more individual users become the audience of cloud storage. With the development of cloud computing for decades, the storage technology in the public cloud environment has been very mature, and many excellent cloud disk products have been born in the commercial environment, but its core technology is closed to the outside. For this public cloud disk system for a large number of individual users, the open source community does not have a sound and reliable overall solution. Based on the above reasons, the key modules and technical points involved in the public cloud disk are studied and designed one by one, and a set of reliable cloud disk system implementation architecture is proposed. Firstly, the principle and technology of the remote file system operation mechanism of cloud disk are analyzed, and the CFSRPC protocol is studied and designed. The specific format definition of client and server and the communication mechanism of the two protocols are given. Through the implementation of the protocol, the cloud disk system can make the details of the remote file operation transparent and facilitate the implementation of the upper layer business of the client and server. At the same time, the transaction mechanism is designed for the protocol to granulate file transfer service, and the CSDA algorithm is designed to dynamically adjust the block size of file transfer at the protocol level, which increases the performance optimization of the cloud disk system. Then the core business process of cloud disk system is analyzed and integrated design for several key modules of cloud disk system is analyzed and integrated. The overall software architecture with access layer management layer and storage layer as the core is summarized. Based on the design of core storage module of Ceph distributed file system, the realization principle of CephFS and MDS is studied and analyzed. According to the definition of cloud disk system architecture and CFSRPC protocol, the CCD. access module of Ceph is designed and implemented with the multichannel IO multiplexing mechanism. The request of cloud disk system is divided into file system operation request and management function operation request, which are received and processed by CCD module and management function module respectively. The load mechanism of adding LVS Keepalived in the middle enables the CCD node and the management function node to support horizontal expansion. An independent authentication system is designed and implemented with reference to the token authentication mechanism of Keystone. According to the definition of CFSRPC protocol, the client is designed and implemented based on NW.js. Finally, the whole cloud disk system is deployed in the test environment, the authentication process and the basic operation of the cloud disk are tested, and the performance of the single point CCD is tested to confirm that the whole system can meet the performance requirements of the public cloud production environment.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333
本文编号：2396151
[Abstract]:Data is one of the core and most valuable products in the Internet age. While consuming and using Internet products, individuals have produced a large number of data that need to be persisted and stored at the same time. The solution of individual storage devices can no longer meet the requirements of individual users under the condition of increasing storage capacity and storage requirements, so more and more individual users become the audience of cloud storage. With the development of cloud computing for decades, the storage technology in the public cloud environment has been very mature, and many excellent cloud disk products have been born in the commercial environment, but its core technology is closed to the outside. For this public cloud disk system for a large number of individual users, the open source community does not have a sound and reliable overall solution. Based on the above reasons, the key modules and technical points involved in the public cloud disk are studied and designed one by one, and a set of reliable cloud disk system implementation architecture is proposed. Firstly, the principle and technology of the remote file system operation mechanism of cloud disk are analyzed, and the CFSRPC protocol is studied and designed. The specific format definition of client and server and the communication mechanism of the two protocols are given. Through the implementation of the protocol, the cloud disk system can make the details of the remote file operation transparent and facilitate the implementation of the upper layer business of the client and server. At the same time, the transaction mechanism is designed for the protocol to granulate file transfer service, and the CSDA algorithm is designed to dynamically adjust the block size of file transfer at the protocol level, which increases the performance optimization of the cloud disk system. Then the core business process of cloud disk system is analyzed and integrated design for several key modules of cloud disk system is analyzed and integrated. The overall software architecture with access layer management layer and storage layer as the core is summarized. Based on the design of core storage module of Ceph distributed file system, the realization principle of CephFS and MDS is studied and analyzed. According to the definition of cloud disk system architecture and CFSRPC protocol, the CCD. access module of Ceph is designed and implemented with the multichannel IO multiplexing mechanism. The request of cloud disk system is divided into file system operation request and management function operation request, which are received and processed by CCD module and management function module respectively. The load mechanism of adding LVS Keepalived in the middle enables the CCD node and the management function node to support horizontal expansion. An independent authentication system is designed and implemented with reference to the token authentication mechanism of Keystone. According to the definition of CFSRPC protocol, the client is designed and implemented based on NW.js. Finally, the whole cloud disk system is deployed in the test environment, the authentication process and the basic operation of the cloud disk are tested, and the performance of the single point CCD is tested to confirm that the whole system can meet the performance requirements of the public cloud production environment.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333
【参考文献】
相关期刊论文 前10条
1 穆彦良;徐振明;;Ceph存储中基于温度因子的CRUSH算法改进[J];成都信息工程学院学报;2015年06期
2 王文睿;;node-webkit:HTML5桌面应用运行环境[J];程序员;2014年01期
3 吉慧;;云存储产品性能测试方法研究[J];电信技术;2013年07期
4 任宇宁;;云计算时代的存储技术——云存储[J];科技传播;2012年03期
5 李杨;李曙东;黄亮;;Samba服务器的研究与应用[J];中国水运(下半月);2011年06期
6 吴一民;刘伟安;;基于Fuse的用户态文件系统的设计[J];微计算机信息;2010年06期
7 王曰芬;章成志;张蓓蓓;吴婷婷;;数据清洗研究综述[J];现代图书情报技术;2007年12期
8 崔滨;万旺根;余小清;楼顺天;;基于EPOLL机制的LINUX网络游戏服务器实现方法[J];微计算机信息;2006年21期
9 李春林,王丽芳,蒋泽军,陈小平;基于身份认证技术安全体系的研究[J];微电子学与计算机;2005年04期
10 史芳丽,周亚莉;Linux系统中虚拟文件系统内核机制研究[J];陕西师范大学学报(自然科学版);2005年01期
相关会议论文 前1条
1 张毕涛;辛阳;;基于Ceph的海量小文件存储的优化方法[A];第十届中国通信学会学术年会论文集[C];2014年
，本文编号：2396151
本文链接：https://www.wllwen.com/kejilunwen/jisuanjikexuelunwen/2396151.html
