一种面向大规模数据密集计算的缓存方法

发布时间：2018-03-28 17:40

本文选题：数据密集计算　切入点：缓存　出处：《计算机研究与发展》2015年07期

【摘要】：随着高性能计算机逐步应用在大规模数据处理领域,存储系统将成为制约数据处理效率的主要瓶颈.在分析了影响数据密集型计算I/O性能若干关键因素的基础上,提出使用计算结点本地存储构建协作式非易失缓存、以分布式存储架构加速集中式存储架构的方法.该方法基于应用层协同使用分布化的本地存储资源,使用非易失存储介质构成大缓存空间,存放大规模数据分析的中间过程结果,以此实现高缓存命中率,并利用并发度约束控制等手段避免I/O竞争,充分利用本地存储的特定性能优势保证缓存加速效果,从而有效地提高了大规模数据处理过程的I/O效率.基于多平台多种I/O模式的测试结果证实了该方法的有效性,聚合I/O带宽具有高扩展性,典型数据密集应用的整体性能最大可提升6倍.
[Abstract]:With the gradual application of high-performance computers in large-scale data processing, storage systems will become the main bottleneck restricting the efficiency of data processing. This paper proposes a method of constructing cooperative non-volatile cache using local storage of computing nodes and accelerating centralized storage architecture with distributed storage architecture. The method is based on the collaborative use of distributed local storage resources in the application layer. Using non-volatile storage media to form a large cache space, storing the intermediate process results of large-scale data analysis, so as to achieve a high cache hit rate, and using concurrency degree constraint control to avoid I / O competition. The specific performance advantage of local storage is fully utilized to ensure the cache acceleration effect, which effectively improves the I / O efficiency of large-scale data processing process. The test results based on multiple I / O modes on multiple platforms demonstrate the effectiveness of the proposed method. The aggregate I / O bandwidth is highly scalable, and the overall performance of typical data-intensive applications can be improved by up to six times.
【作者单位】：高性能计算国家重点实验室(国防科学技术大学);中国石油集团东方地球物理勘探公司;
【基金】：国家自然科学基金项目(61120106005) 国家“八六三”高技术研究发展计划基金项目(2012AA01A301)
【分类号】：TP333

【参考文献】