基于对象存储的分布式小文件系统的研究与设计

发布时间：2018-03-10 11:20

本文选题：分布式文件系统　切入点：对象存储　出处：《清华大学》2012年硕士论文　论文类型：学位论文

【摘要】：随着互联网的蓬勃发展，云计算的浪潮汹涌澎湃，越来越多的互联网企业选择将他们的业务迁入云中，伴随业务的发展，研究人员发现来源于博客，社区，视频网站，微博等数据密集型web2.0网络应用，会在应用后台产生数目庞大的小文件，而且随着时间的推移，海量小文件的增长呈几何级增长，同时，小文件的存储访问管理也带来了诸如高并发，大流量，以及热点不集中等问题，传统的分布式文件系统如NAS等，并不能有效应对这种海量增长的小文件服务应用。为了应对海量小文件高并发，高频度，大流量，以及存储管理的问题，本文提出了基于对象存储的分布式文件系统BerryStore的设计与实现。通过对Google，FaceBook，淘宝等公司提出的成熟的分布式文件系统解决方案的研究后，BerryStore结合了以上几种方案的优势之处，通过采用对象存储的方式将海量的小文件集合成大对象文件进行存储管理，，并在内存中建立对象文件元数据索引的方式，减少了小文件访问在系统中的IO开销。我们进一步利用Zookeeper设计并构建集群的协调控制体系结构，实现了对整个分布式系统进行配置，状态，数据同步以及管理的功能，避免了复杂的数据一致性访问控制流程，简化了整个存储系统的复杂度，提高了该小文件系统的可靠性。本文首先介绍目前主流的分布式文件系统在云计算平台中的应用现状，并对若干经典的架构技术进行了研究，在对比这些已有的方案之后，我们设计了应对海量小文件存储的一个系统方案，并在后面对方案进行了详细设计，对该原型系统进行实现以及性能评测，并将其与之前我们所探讨的已有方案进行对比。最后我们对目前阶段的工作进行了总结并对未来工作进行了展望。
[Abstract]:With the rapid development of the Internet, cloud computing is surging. More and more Internet enterprises choose to move their business into the cloud. With the development of business, researchers find that it comes from blogs, communities, video sites. Data-intensive web2.0 network applications such as Weibo will produce a large number of small files in the application background, and with the passage of time, the growth of massive small files will show a geometric growth, at the same time, Storage and access management of small files also brings problems such as high concurrency, large traffic, and not focusing on hot spots. Traditional distributed file systems, such as NAS, can not effectively cope with the huge growth of small file service applications. To deal with large numbers of small files with high concurrency, high frequency, large traffic, and storage management problems, This paper presents the design and implementation of distributed file system (BerryStore) based on object storage. By using the method of object storage, the large amount of small files are assembled into large object files for storage and management, and the object file metadata index is built in memory. We further use Zookeeper to design and construct the coordinated control architecture of the cluster, and realize the functions of configuration, state, data synchronization and management of the whole distributed system. The complex data consistency access control flow is avoided, the complexity of the whole storage system is simplified, and the reliability of the small file system is improved. This paper first introduces the current application status of the mainstream distributed file system in cloud computing platform, and studies some classical architecture technologies. After comparing these existing solutions, We design a system scheme to store large amount of small files, and then design the scheme in detail, implement the prototype system and evaluate the performance of the prototype system. Finally, we summarize the current stage of the work and look forward to the future work.
【学位授予单位】：清华大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【相似文献】