融合多核和MapReduce的连接聚集查询优化

发布时间：2018-08-23 20:55

【摘要】：连接聚集查询是大规模数据分析的核心操作算子之一,多核处理器为大规模数据的连接聚集查询操作提供了优化空间但同时也存在很大挑战.主要针对多核处理器集群环境,研究MapReduce框架下大规模数据连接聚集查询的优化算法.首先,基于传统MapReduce框架设计并实现了Map端的单线程连接聚集查询算法,基于实验分析指出MapReduce和多核处理器融合提升性能的必要性;其次,针对处理器的多核架构,设计并实现了Map端的多线程连接聚集查询算法,找出了MapReduce和多核处理器融合的性能瓶颈;进而,提出了Map端对输入分片无竞争读取的多线程连接聚集查询算法,该算法让MapReduce充分利用了多核处理器的性能优势.实验结果表明,提出的算法在大规模数据的连接聚集查询处理上充分发挥了硬件优势,具有较好的时间性能和可扩展性.
[Abstract]:Join aggregation query is one of the core operators in large-scale data analysis. Multi-core processors provide optimization space for large scale data join aggregation query operation, but there are also great challenges at the same time. Aiming at multi-core processor cluster environment, the optimization algorithm of large scale data join aggregation query under MapReduce framework is studied. Firstly, a single thread join aggregation query algorithm based on traditional MapReduce framework is designed and implemented. Based on experimental analysis, the necessity of MapReduce and multi-core processor fusion to improve performance is pointed out. This paper designs and implements a multi-thread join aggregation query algorithm in Map, finds out the performance bottleneck of MapReduce and multi-core processor fusion, and then proposes a multi-thread join aggregation query algorithm based on Map. This algorithm allows MapReduce to take full advantage of the performance advantages of multi-core processors. The experimental results show that the proposed algorithm can give full play to the hardware advantages and has good time performance and extensibility in large-scale data join aggregation query processing.
【作者单位】：桂林电子科技大学广西可信软件重点实验室;桂林电子科技大学广西自动检测技术与仪器重点实验室;
【基金】：国家自然科学基金项目(61462017;61363005) 广西自然科学基金项目(2014GXNSFAA118353;2014GXNSFAA118390;2014GXNSFDA118036) 广西高校科研资助项目(2013YB083) 广西自动检测技术与仪器重点实验室基金项目(YQ15110;YQ14109) 桂林电子科技大学研究生创新资助项目(GDYCSZ201465) 广西高等学校高水平创新团队及卓越学者计划
【分类号】：TP332

【参考文献】