基于性能预测的Spark资源优化分配策略

发布时间：2018-11-03 16:17

【摘要】：Spark已经成为如今最流行的分布式大数据计算平台,由于其高效的性能、良好的容错性与统一性,在业界得到了广泛的使用。但由于Spark平台对数据的具体操作对于用户来说是透明的,Spark上运行的任务受到许多因素的影响,比如数据的分区策略,算法的设计与实现以及节点的资源分配等等。使得对Spark性能预测的非常困难。本课题通过建立基于Spark任务结构的性能模型,研究Spark任务在不同数据量、分区策略的情况下的执行时间,并在此基础上寻找任务执行时间与集群资源消耗间的平衡,提出基于动态重分区的资源分配优化策略。本文在细粒度监控集群资源的基础上,解析Spark任务各阶段的执行信息,建立基于Spark任务结构的性能模型,通过大量的历史实验数据训练模型参数,实现了对不同负载类型Spark计算任务的性能预测。在此基础上,我们研究了分区策略对Spark执行时间的影响,我们发现尽管增加节点的并行度可以在一定程度上提升计算任务的性能,但在一些情况下,性能提升的幅度与新增的资源消耗相比起来,可以认为是微乎其微的,当我们已经满足了用户在任务运行时间方面的需求,这些微小的性能提升便可以忽略,相应的,我们应该在用户给出的时间要求下尽可能的减少资源配置,以达到节约资源的目的。我们将会通过在一系列的实际Spark计算任务中加入动态重分区的方式寻找任务的最佳分区方案,提出基于任务时间预测的重分区策略。在不过多牺牲任务运行时间的前提下,节约集群资源,找到任务执行时间与集群资源配置的平衡,指导用户对Spark任务合理使用集群资源。本文通过实验验证了性能模型的合理性与对任务执行时间预测的准确性,取得了不错的预测准确性。在此基础上我们提出基于性能预测的资源优化分配策略,在Spark负载集合中通过动态重分区的方法,寻找优化的集群资源分配策略,以取得任务执行时间与集群资源消耗间的平衡。实验结果表明,我们的优化策略可以用户给出的执行时间内较为明显地节约集群资源,在任务执行时间与集群资源消耗之间寻找到了良好的平衡。
[Abstract]:Spark has become the most popular distributed big data computing platform. Because of its high performance, good fault tolerance and unity, it has been widely used in the industry. However, because the operation of Spark platform is transparent to users, the tasks running on Spark are affected by many factors, such as data partitioning strategy, algorithm design and implementation, resource allocation of nodes and so on. This makes it very difficult to predict Spark performance. By establishing a performance model based on Spark task structure, this paper studies the execution time of Spark task under different data volume and partition strategy, and then finds out the balance between task execution time and cluster resource consumption. An optimal resource allocation strategy based on dynamic repartitioning is proposed. On the basis of fine-grained monitoring cluster resources, this paper analyzes the execution information of each stage of Spark task, establishes a performance model based on Spark task structure, and trains the parameters of the model through a large number of historical experimental data. The performance prediction of Spark computing task with different load types is realized. On this basis, we study the effect of partitioning policy on the execution time of Spark. We find that although increasing the degree of parallelism of nodes can improve the performance of computing tasks to some extent, in some cases, The performance improvement is considered to be minimal compared with the additional resource consumption, and when we have met the user's requirements for task runtime, these small performance improvements can be ignored. In order to save resources, we should reduce the allocation of resources as much as possible under the time requirement given by the user. We will find the best partitioning scheme by adding dynamic repartitioning to a series of actual Spark computing tasks and propose a repartitioning strategy based on task time prediction. On the premise of not sacrificing task running time too much, we can save cluster resources, find the balance between task execution time and cluster resource allocation, and guide users to use cluster resources reasonably for Spark tasks. The rationality of the performance model and the accuracy of the prediction of task execution time are verified by experiments in this paper. On this basis, we propose an optimal resource allocation strategy based on performance prediction, and find the optimized cluster resource allocation strategy through dynamic repartitioning in the Spark load set. To achieve a balance between task execution time and cluster resource consumption. The experimental results show that our optimization strategy can obviously save cluster resources in the execution time given by users and find a good balance between task execution time and cluster resource consumption.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【相似文献】