Squid小文件缓存优化的设计与实现

发布时间：2018-07-14 15:26

【摘要】：提高网页浏览速度、提高用户体验是所有网络服务商面临的共同挑战和长期的努力方向。代理服务器作为改善网络服务质量的基本手段之一,受到业界的广泛关注和重视。 Squid是一款典型的代理服务器,支持HTTP、FTP等数据对象的代理,支持多样化访问控制以及多种操作系统平台,是全球范围内使用最广泛的代理服务器。Squid使用网页缓存来减少网络带宽并提高响应速度。本文针对Squid中的小文件的缓存进行了深入研究,设计并实现了能有效识别热点数据,提高缓存命中率的算法并设计实现软件模块模块。论文的主要工作如下: (1)本文的研究背景主要包括:分析研究提高网络服务的方法,介绍代理服务器,研究已有的小文件存储系统并总结优化海量小文件存储的方法。 (2)分析Squid的体系结构和实现原理,尤其是对Coss存储模块的研究,提出该系统在小文件存储中存在的关键问题。 (3)设计并实现了对冷热数据分级缓存的方案,描述并实现了Relocation Cache方法。 (4)分析了分级缓存中如何识别热点数据的关键问题,比较几种识别热点数据的方案之后,提出自适应滑动窗口机制,通过实现该方案来进一步提升系统性能。 (5)选择网络代理服务中几个关键的benchmark,分别对系统未优化之前、第一次优化以及第二次优化之后的性能进行测试分析。测试表明优化后的Coss存储系统在面对大量小文件的存储访问时,能更好的缓存热点数据,从而降低磁盘I/O操作,提高系统的整体性能。本论文的研究成果对于其它的海量小文件存储系统的缓存设计也具有较好的借鉴价值。
[Abstract]:Improving web browsing speed and user experience is a common challenge and long-term direction for all web service providers. As one of the basic methods to improve the quality of network service, the proxy server is paid more and more attention by the industry. Squid is a typical proxy server, which supports the proxy of data objects such as HTTP and FTP. Supported by diversified access control and multiple operating system platforms, Squid is the most widely used proxy server in the world. Squid uses web caching to reduce network bandwidth and improve response speed. This paper studies the cache of small files in Squid, designs and implements an algorithm to identify hot data, improve the hit rate of cache, and design and implement software module. The main work of this paper is as follows: (1) the research background of this paper mainly includes: analyzing the methods of improving network service, introducing proxy server, The existing small file storage system is studied and the methods to optimize the mass small file storage are summarized. (2) the structure and implementation principle of Squid are analyzed, especially the research on Coss storage module. The key problems of the system in small file storage are put forward. (3) A scheme of grading buffer for cold and hot data is designed and implemented. This paper describes and implements the relocation cache method. (4) after analyzing the key problems of how to identify hot data in hierarchical cache and comparing several schemes to identify hot data, an adaptive sliding window mechanism is proposed. Through the implementation of the scheme to further improve the system performance. (5) selected several key network agent service benchmark, respectively before the system is not optimized, the first optimization and the second optimization after the performance of the test analysis. The test results show that the optimized Coss storage system can cache hot data better in the presence of a large number of small files, thus reducing the disk I / O operation and improving the overall performance of the system. The research results of this paper also have good reference value for the cache design of other mass small file storage systems.
【学位授予单位】：上海交通大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【相似文献】