基于学习的容器环境Spark性能监控与分析

发布时间：2018-03-29 09:44

本文选题：Spark　切入点：容器　出处：《计算机应用》2017年12期

【摘要】：Spark计算框架被越来越多的企业用作大数据分析的框架,由于通常部署在分布式和云环境中因此增加了该系统的复杂性,对Spark框架的性能进行监控并查找导致性能下降的作业向来是非常困难的问题。针对此问题,提出并编写了一种针对分布式容器环境中Spark性能的实时监控与分析方法。首先,通过在Spark中植入代码和监控Docker容器中的API文件获取并整合了作业运行时资源消耗信息;然后,基于Spark作业历史信息,训练了高斯混合模型(GMM);最后,使用训练后的模型对Spark作业的运行时资源消耗信息进行分类并找出导致性能下降的作业。实验结果表明,所提方法能检测出90.2%的异常作业,且其对Spark作业性能的影响仅有4.7%。该方法能减轻查错的工作量,帮助用户更快地发现Spark的异常作业。
[Abstract]:The Spark computing framework is used by more and more enterprises as the framework for big data analysis, which adds to the complexity of the system because it is usually deployed in distributed and cloud environments. It has always been very difficult to monitor the performance of Spark framework and find jobs that lead to performance degradation. In order to solve this problem, a real-time monitoring and analysis method for Spark performance in distributed container environment is proposed and written. Through embedding code in Spark and monitoring API files in Docker container, the information of resource consumption at job run time is obtained and integrated. Then, Gao Si mixed model is trained based on Spark job history information. Using the trained model to classify the resource consumption information of the Spark jobs and find out the jobs that lead to the deterioration of the performance. The experimental results show that the proposed method can detect 90.2% abnormal jobs, and the results of the experiments show that the proposed method can detect 90.2% of the abnormal jobs. This method can reduce the workload of error detection and help users to find abnormal jobs of Spark more quickly.
【作者单位】：同济大学计算机科学与技术系;嵌入式系统与服务计算教育部重点实验室(同济大学);
【分类号】：TP18;TP277

【相似文献】