基于HDFS的云存储系统小文件优化方案

发布时间：2018-03-18 22:38

本文选题：Hadoop分布式文件系统　切入点：小文件　出处：《计算机工程》2016年03期 　论文类型：期刊论文

【摘要】：Hadoop分布式文件系统(HDFS)具有高容错、可伸缩、廉价存储等优良特性,在大数据存储和分析场景中得到广泛应用。但对于海量小文件存储,HDFS存在高内存消耗、高延迟访问等缺陷。为此,结合"合肥城市云"系统"一次上传,多次下载"的特性,提出一种基于小文件属性的优化方案。根据文件之间的相关性设定优先级,对小于5 MB的文件按优先级高低合并后再上传,并生成索引记录。结合随机化思想,采用两级缓存策略,将预提取数据缓存在内存池中,提高访问效率。同时,系统定期查询访问日志,根据用户访问习惯,动态调整预提取因子的大小。实验结果表明,该方案能有效提高小文件访问效率,降低名字节点和数据节点的内存开销,在有海量小文件存取的情况下提升系统的交互性。
[Abstract]:Hadoop (distributed File system) has been widely used in big data storage and analysis scenarios because of its high fault tolerance, scalability and cheap storage, but it has high memory consumption for large amount of small file storage. For this reason, combined with the feature of "once upload, multiple download" in Hefei Urban Cloud system, an optimization scheme based on the attributes of small files is proposed, and the priority is set according to the correlation between files. The files with less than 5 MB are merged according to the priority and then uploaded, and the index records are generated. Combined with the idea of randomization, the pre-extracted data is cached in the memory pool by using the two-level cache strategy, and the access efficiency is improved. The system regularly queries the access log and dynamically adjusts the size of the pre-extraction factor according to the user's visiting habits. The experimental results show that the proposed scheme can effectively improve the access efficiency of small files and reduce the memory overhead of name nodes and data nodes. In the case of large amounts of small file access to enhance the system's interactivity.
【作者单位】：中国科学技术大学自动化系;
【基金】：国家自然科学基金资助项目(61174062)
【分类号】：TP333

【相似文献】