非CDN环境小文件存取优化研究

发布时间：2018-07-09 14:36

本文选题：海量小文件 + 分布式文件系统　；参考：《昆明理工大学》2017年硕士论文

【摘要】：现阶段,随着企业业务增长,数据存储量高速增长,企业对于数据存储的可扩展性不断提出了的要求。在新存储需求下,传统的数据存储方式已经无法满足新一代的技术要求,扩展性强、价格低廉的分布式存储系统则逐渐成为众多企业的选择。当前市面上主要的分布式存储产品,要么为大型对象存储设计,要么是基于互联网的产品,并不适用于没有CDN(Content Delivery Network)的企业内部环境中。本文在weedfs的基础上,以分布式存储技术为核心,基于企业对小文件存储需求的现状,对无CDN环境下的小文件分布存储应用进行了初步探索,具体开展了以下研究:第一,回顾存储系统的发展历程,分析当前各个分布存储技术的特点。第二,为weedfs添加了一个分布式缓存系统,并针对传统缓存算法的缺点,改进了缓存算法对各种场景的适应能力。第三,结合传统的缓存算法和机器学习技术,进一步提高整个缓存系统的性能。本文通过引入一个缓存系统对weedfs进行了改进,同时将机器学习和传统的缓存技术相结合,为企业内部处理海量小文件提供了一个新的选择。本研究对其他行业存储系统的选择和缓存算法的改进也具有一定的借鉴意义。
[Abstract]:At present, with the growth of enterprise business and the rapid growth of data storage capacity, the scalability of data storage is constantly required by enterprises. Under the new storage demand, the traditional data storage method can not meet the new generation of technical requirements. The distributed storage system with strong expansibility and low price has gradually become the choice of many enterprises. At present, the main distributed storage products in the market are either designed for large-scale object storage or Internet-based products, and are not suitable for the internal environment of enterprises without CDN (content delivery Network). On the basis of weedfs, based on the distributed storage technology and the current situation of enterprises' demand for small file storage, this paper makes a preliminary exploration on the application of distributed small file storage in the non-CDN-based environment. The following researches are carried out: first, This paper reviews the development of storage system and analyzes the characteristics of each distributed storage technology. Secondly, we add a distributed cache system to weedfs, and improve the adaptability of the cache algorithm to various scenarios in view of the shortcomings of the traditional cache algorithm. Thirdly, the performance of the whole cache system is further improved by combining the traditional cache algorithm and machine learning technology. This paper improves weedfs by introducing a cache system, and combines machine learning with traditional cache technology, which provides a new choice for internal processing of large amount of small files. This research also has certain reference significance to other industries storage system choice and the cache algorithm improvement.
【学位授予单位】：昆明理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【相似文献】