一种面向HDFS的数据随机访问方法
发布时间：2019-05-22 04:48
【摘要】：为了简化文件系统的实现,支持超大规模数据集的流式访问,HDFS牺牲了文件的随机访问功能,而在实际场景中很多应用都需要对文件进行随机访问。在深入分析HDFS数据读写原理的基础上,提出了一种面向HDFS的数据随机访问方法。其设计思想是为Datanode添加本地数据访问接口,用户程序可以读取Datanode上存放的数据块文件以及把数据写入到Datanode上的数据块存放目录。文件的首副本由用户程序直接产生,其余副本在首副本写入完成之后采用数据复制的方式生成。此外,为数据块添加了权限管理功能,Datanode上的文件副本属于用户所有。若名字空间中文件权限发生变化,文件对应的数据块权限也会改变。测试表明,数据读取性能提升了约10%,数据写入性能提升了20%以上,在高并发下写入性能最大可提升2.5倍。
[Abstract]:In order to simplify the implementation of file system and support streaming access to super-large data sets, HDFS sacrifices the random access function of files, and many applications need random access to files in practical scenarios. Based on the deep analysis of the principle of HDFS data reading and writing, a HDFS oriented data random access method is proposed. The design idea is to add a local data access interface for Datanode. The user program can read the data block files stored on the Datanode and write the data to the data block storage directory on the Datanode. The first copy of the file is generated directly by the user program, and the other copies are generated by data replication after the first copy is written. In addition, permission management function is added to the data block, and the file copy on Datanode belongs to the user. If the file permissions in the name space change, the data block permissions corresponding to the file will also change. The test results show that the data reading performance is improved by about 10%, the data writing performance is improved by more than 20%, and the write performance can be improved by 2.5 times at high concurrency.
【作者单位】： 中国科学院高能物理研究所;中国科学院大学;
【基金】：国家自然科学基金(No.11375223,No.11375221)
【分类号】：TP311.13;TP333
本文编号：2482690
[Abstract]:In order to simplify the implementation of file system and support streaming access to super-large data sets, HDFS sacrifices the random access function of files, and many applications need random access to files in practical scenarios. Based on the deep analysis of the principle of HDFS data reading and writing, a HDFS oriented data random access method is proposed. The design idea is to add a local data access interface for Datanode. The user program can read the data block files stored on the Datanode and write the data to the data block storage directory on the Datanode. The first copy of the file is generated directly by the user program, and the other copies are generated by data replication after the first copy is written. In addition, permission management function is added to the data block, and the file copy on Datanode belongs to the user. If the file permissions in the name space change, the data block permissions corresponding to the file will also change. The test results show that the data reading performance is improved by about 10%, the data writing performance is improved by more than 20%, and the write performance can be improved by 2.5 times at high concurrency.
【作者单位】： 中国科学院高能物理研究所;中国科学院大学;
【基金】：国家自然科学基金(No.11375223,No.11375221)
【分类号】：TP311.13;TP333
【相似文献】
相关期刊论文 前4条
1 ;室温超低电压操作的高密度磁电阻随机访问存储器[J];中国基础科学;2012年01期
2 熊青玲;同步静态随机访问存储器常见问题解析[J];电子器件;2004年01期
3 孙健;陈岚;郝晓冉;;基于PCRAM主存系统的访问机制[J];微电子学与计算机;2014年01期
4 ;[J];;年期
相关会议论文 前1条
1 代芬;王卫星;俞龙;;同步静态随机访问存储器的特点及应用[A];农业工程科技创新与建设现代农业——2005年中国农业工程学会学术年会论文集第三分册[C];2005年
相关硕士学位论文 前2条
1 赵琨;云存储中支持隐私保护的隐藏性随机访问机制研究[D];电子科技大学;2013年
2 朱婷;大容量静态随机访问存储器的低功耗研究[D];电子科技大学;2011年
，本文编号：2482690
本文链接：https://www.wllwen.com/kejilunwen/jisuanjikexuelunwen/2482690.html
