基于HBase的小文件高效存储方法

发布时间：2018-06-25 08:00

本文选题：Hadoop分布式文件系统(HDFS) + 海量数据　；参考：《重庆邮电大学学报(自然科学版)》2016年01期

【摘要】：基于Hadoop平台的相关系统得到了广泛应用。Hadoop分布式文件系统(Hadoop distributed file system,HDFS)通过分布式的工作方式,负责处理海量文件数据。对HDFS而言,海量数据中的小文件存储问题制约着系统高效工作的能力。针对海量数据中小文件读写效率低的情况,提出一种基于HBase(Hadoop database)的海量小文件高效存储方法,利用HBase的存储优势,将小文件直接存储于HBase,从而有效减少元数据节点服务器(NameNode)的负载,并对上层应用系统提供透明的访问接口。实验结果表明,该方法可以实现海量小文件的高效存储,提高HDFS环境下小文件的读写效率。
[Abstract]:The related systems based on Hadoop platform have been widely used. Hadoop distributed file system (HDFs) is responsible for dealing with large amount of file data through distributed working mode. For HDFS, the problem of small file storage in massive data restricts the system's ability to work efficiently. In view of the low reading and writing efficiency of small and medium files in mass data, a high efficient storage method based on HBase (Hadoop database) is proposed, which makes use of the storage advantage of HBase. Small files are stored in HBasedirectly, which effectively reduces the load of metadata node server (name Node) and provides a transparent access interface to the upper application system. The experimental results show that this method can realize the efficient storage of large amount of small files and improve the efficiency of reading and writing of small files in HDFS environment.
【作者单位】：重庆邮电大学计算机科学与技术学院;
【分类号】：TP333

【相似文献】