改进的Spark Shuffle内存分配算法

发布时间：2018-04-10 02:27

本文选题：Apache　切入点：Spark　出处：《计算机应用》2017年12期

【摘要】：Shuffle性能是影响大数据集群性能的重要指标,Spark自身的Shuffle内存分配算法试图为内存池中的每一个Task平均分配内存,但是在实验中发现,由于各Task对于内存需求的不均衡导致了内存的浪费和运行效率较低的问题。针对上述问题,提出一种改进的Spark Shuffle内存分配算法。该算法根据Task的内存申请量和历史运行数据将Task按内存需求分为大小两类,对小内存需求型Task作"分割化"处理,对大内存需求型Task基于Task溢出次数和溢出后等待时间分配内存。该算法充分利用内存池的空闲内存,可以在数据倾斜导致的Task内存需求不均衡的情况下进行Task内存分配的自适应调节。实验结果表明,改进后算法较原算法降低了Task的溢出率,减少了Task的周转时间,提高了集群的运行性能。
[Abstract]:Shuffle performance is an important indicator affecting big data's cluster performance. Spark's own Shuffle memory allocation algorithm tries to allocate memory equally for each Task in the memory pool, but it is found in the experiment.Due to the imbalance of memory requirements in Task, memory is wasted and inefficient.To solve the above problems, an improved Spark Shuffle memory allocation algorithm is proposed.According to the memory application amount and the historical running data of Task, the Task is divided into two categories according to the memory requirement, and the small memory demand type Task is "partitioned".For large memory Task based on Task overflow times and after overflow wait time allocation of memory.This algorithm makes full use of the free memory of memory pool and can adjust the Task memory allocation adaptively under the condition that the demand of Task memory caused by data skew is not balanced.The experimental results show that compared with the original algorithm, the improved algorithm reduces the overflow rate of Task, reduces the turnover time of Task, and improves the performance of the cluster.
【作者单位】：中国民航大学计算机科学与技术学院;
【基金】：国家自然科学基金资助项目(U1533104)~~
【分类号】：TP301.6;TP333

【相似文献】