改善Hadoop文件处理效率的技术研究

发布时间：2018-06-04 16:32

本文选题：分布式文件系统 + 小文件　；参考：《微电子学与计算机》2014年07期

【摘要】：提出一种改善Hadoop文件处理效率的方法,在Hadoop中添加一个小文件处理模块SFPM,根据文件名为海量小文件建立二级索引,同时采用预加载技术将索引提前存入缓存,可提高文件查找访问效率;在合并文件时,采取舍弃多余空间的策略,避免将一个文件拆分存储在两个block上,减少了文件访问时间开销.实验结果表明该方法能有效减轻NameNode的负荷,提高小文件读写效率.
[Abstract]:A method to improve the efficiency of Hadoop file processing is put forward. A small file processing module, SFPM, is added to the Hadoop. According to the name of the file, a two level index is set up for a large number of small files. At the same time, the preload technology is used to save the index into the cache in advance. It can improve the efficiency of the file search access. It is avoided to save a file on two block and reduce the time cost of file access. Experimental results show that the method can effectively reduce the load of NameNode and improve the efficiency of reading and writing of small files.
【作者单位】：广东工业大学计算机学院;
【基金】：广东省战略性新兴产业核心技术攻关项目(2012A010701004) 广东省自然科学基金重点项目(S2012020011071) 广东省教育部产学研合作项目(2012B091000037,2012B091000041)
【分类号】：TP316.4;TP333

【参考文献】