一种面向HDFS的多层索引技术

发布时间：2018-02-02 14:58

本文关键词： 查询索引 HDFS SOH 大数据　出处：《软件学报》2017年03期 　论文类型：期刊论文

【摘要】：SOH(SQL over HDFS)系统通常将数据存储于分布式文件系统HDFS(Hadoop distributed file system)中,采用Map/Reduce或分布式查询引擎来处理查询任务.得益于HDFS以及Map/Reduce的容错能力和可扩展性,SOH系统可以很好地应对数据规模的飞速增长,完成分析型查询处理.然而,在处理选择型查询或交互式查询时,这类系统暴露出了性能上的缺陷.提出一种通用的索引技术,可以应用于SOH系统中,以提高其查询处理的效率.分析了SOH系统访问HDFS文件的过程,指出了其中影响数据加载时间的关键因素.提出了split层和split内部双层索引机制;设计并实现了聚集索引和非聚集索引;最后,在标准数据集上进行了大量实验,并与现有基于HDFS的索引技术进行了比较.实验结果表明,所提出的索引技术可以有效地提高查询处理的效率.
[Abstract]:SOH(SQL over HDFS systems typically store data in a distributed file system, HDFS (. In the Hadoop distributed file system. Use Map/Reduce or distributed query engine to handle query tasks, thanks to the fault tolerance and extensibility of HDFS and Map/Reduce. SOH system can well cope with the rapid growth of data scale, complete analytical query processing. However, when dealing with selective queries or interactive queries. This kind of system exposes the performance defect. A general indexing technique is proposed, which can be applied to SOH system. In order to improve the efficiency of query processing, the process of accessing HDFS files in SOH system is analyzed. The key factors affecting data loading time are pointed out, and the split layer and split internal double-layer indexing mechanism are proposed. The clustered index and the nonclustered index are designed and implemented. Finally, a large number of experiments are carried out on the standard data set and compared with the existing indexing techniques based on HDFS. The experimental results show that the proposed indexing technology can effectively improve the efficiency of query processing.
【作者单位】：中国人民大学信息学院;教育部数据工程与知识工程重点实验室(中国人民大学);
【基金】：国家重点研发计划(2016YFB1000702) 中国人民大学预研委托(团队)基金(14XNLQ06) 国家自然科学基金(61003086)~~
【分类号】：TP311.13
【正文快照】： 2(教育部数据工程与知识工程重点实验室(中国人民大学),北京100872)1000-9825/5161.htm英文引用格式:He L,Chen JC,Du XY.Multi-Layered index for HDFS-based systems.Ruan Jian Xue Bao/Journal of Software,2017,28(3):502?513(in Chinese).http://www.jos.org.cn/1000-982

【相似文献】