基于FastDFS的轻量级分布式文件系统的设计与实现

发布时间：2018-06-17 08:27

本文选题：分布式文件系统 + FastDFS　；参考：《东北大学》2013年硕士论文

【摘要】：随着互联网的迅猛发展,信息呈现出爆炸式的增长,传统的文件存储方式已经不能满足系统对于存储空间和处理时间的要求。高性能的分布式文件系统成为相关领域的研究热点。 FastDFS是一个开源的分布式文件系统,FastDFS的文件命名方式和分组方式等机制使得它具有轻量级、高扩展性等优点。在国内外学者已有研究工作的基础上,基于开源分布式文件系统FastDFS,本文针对如今互联网应用的特点,为要求高实时性的互联网应用量身定制了一款分布式文件系统,相比于FastDFS更适合大量文件的存储,特别是短时间内需要响应大量请求的应用。本文主要对FastDFS在以下方面进行了改进：一是同组storage server间同步机制的优化,由原来的根据binlog定时同步更新优化为上传文件的同时进行文件的同步；二是加入缓存层,因为对于互联网应用,本次访问的图片、视频等资源很有可能在短时间内继续被访问。三是对大文件进行分片存储,分片存储方式不仅使系统可以存储更大的文件,而且如果在某台机器宕机或者新增加存储服务器的情况下,只需要同组其他机器同步过来一个文件块的情况下,本机就可对外提供该文件的下载服务,相比于同步整个文件可以更快的对外提供服务,提高系统的可用性和吞吐量。基于VMware虚拟机和Ubuntu11.04操作系统,对本文提出的方案进行了实验测试,与现在FastDFS的最新版本V4.06相比,本文设计的一系列机制能够更快的响应客户端的请求,具有更高的实时性,更好的满足了如今互联网应用对文件操作的需求。
[Abstract]:With the rapid development of the Internet and the explosive growth of information, the traditional file storage method can not meet the requirements of the system for storage space and processing time. FastDFS is an open source distributed file system. FastDFS has the advantages of lightweight and high scalability due to its mechanism of naming and grouping FastDFS. Based on the research work of scholars at home and abroad, and based on open source distributed file system FastDFS, according to the characteristics of Internet application today, this paper customizes a distributed file system for high real-time Internet application. Compared with FastDFS, it is more suitable for storing a large number of files, especially for applications that need to respond to a large number of requests in a short time. This paper mainly improves FastDFS in the following aspects: the first is the optimization of synchronization mechanism between the same group of storage server, which is optimized from the original synchronization update according to binlog timing to the synchronization of files uploaded at the same time, the second is the addition of cache layer. For Internet applications, this visit of images, video and other resources are likely to be accessed in a short time. The third is to divide large files into pieces, which not only allow the system to store larger files, but also if a machine is down or a new storage server is added. If only the same group of other machines synchronize a file block, the local machine can provide the download service of the file. Compared with synchronizing the whole file, it can provide the service faster and improve the availability and throughput of the system. Based on VMware virtual machine and Ubuntu 11.04 operating system, the proposed scheme is tested experimentally. Compared with the latest version of FastDFS, V4.06, a series of mechanisms designed in this paper can respond to client requests more quickly and have higher real-time performance. Better meet the needs of Internet applications for file operations today.
【学位授予单位】：东北大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP316.4;TP333

【参考文献】