多个云环境下大数据工作流性能优化算法的设计与实现

发布时间：2018-02-20 22:28

本文关键词： 工作流映射云计算性能优化大数据　出处：《西北大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着云计算技术的不断发展,融合多个云计算环境的混合云越来越得到广泛的应用。在许多科研应用程序中,要得到最终结果需要执行一系列任务。而这些任务,如数据生成、处理、分析等,之间存在着相互依存的关系,这些任务组合及其相互依存关系构成了工作流。在很多具体的大数据应用实践中,工作流技术已经成为不可或缺的一部分。目前云计算服务分为三种:软件即服务(Software as a Service,SaaS),平台即服务(Platform as a Service,PaaS),基础设施即服务(Infrastructure as a Service,IaaS)。基础设施即服务是将计算资源抽象成一个个虚拟机,将虚拟硬件资源提供给用户。它对于大数据工作流的执行是最适用的。对于一个工作流,可以直接应用云平台所提供的虚拟硬件资源来执行各个任务,就像传统的运行在集群上的工作流一样。在实际应用中,预算约束和可靠性约束都是对大数据工作流的部署非常重要的约束。预算约束是保证工作流能以较低的成本运行完成,而可靠性约束针对的是真实的云环境中的各种不确定性和故障。本文针对在预算和可靠性双重约束条件下的多个云环境中工作流映射问题,抽象出问题的数学模型,分析了问题的复杂性,设计了启发式算法,并进行了实验比较。文章证明了问题的复杂度是属于NP-Complete的,并且通过实验发现本文提出的算法相比较之前的算法有明显的优越性。本文对于工作流映射的研究分为以下几部分:问题的提出与复杂性的证明。本文针对预算受约束,通过工作流映射,建立了针对两种不同可靠性要求下的端对端延迟最小化问题的数学模型。通过复杂性分析,证明了这两个问题都是一类NP-Complete问题的子问题,从而得到这两个问题也都是NP-Complete 的。问题的分析与算法的设计。针对这两个问题,分别提出了启发式算法RMCWM和EMCWM,算法分为虚拟机选择、物理机及带宽选择两个阶段。由于问题中对可靠性的建模采用了 3GG模型,在分析该模型特性的基础上,发现了优化可靠性的规律。并将这种规律融入算法的设计中,取得了很好的效果。模拟实验和性能评估。本文对所设计的算法针对不同规模的云和不同数量的云组合,进行了模拟实验。实验证明了不论在小规模的云组合的情况下还是大规模的云组合的情况下,该算法对比之前的算法都有明显的优越性。
[Abstract]:With the continuous development of cloud computing technology, the hybrid cloud, which integrates multiple cloud computing environments, is becoming more and more widely used. In many scientific applications, a series of tasks, such as data generation, are required to achieve the final results. Processing, analysis, and so on, there are interdependent relationships between these tasks and their interdependence constitute a workflow. In many specific big data application practice, Workflow technology has become an integral part. Cloud computing services are currently divided into three categories: software as a service, platform as a service, infrastructure as a service, infrastructure a service, IaaSan. Infrastructure as a service is to be computed. Resources are abstracted into virtual machines, Provide virtual hardware resources to users. It is most suitable for the execution of big data workflow. For a workflow, you can directly apply the virtual hardware resources provided by the cloud platform to perform various tasks. In practical applications, both budget constraints and reliability constraints are very important constraints for the deployment of big data workflow. Budget constraints are to ensure that the workflow can be completed at a lower cost. The reliability constraint is aimed at all kinds of uncertainties and faults in the real cloud environment. In this paper, the mathematical model of the problem is abstracted from the workflow mapping problem in multiple cloud environments under the condition of double constraints of budget and reliability. The complexity of the problem is analyzed, the heuristic algorithm is designed, and the experimental results are compared. It is proved that the complexity of the problem belongs to NP-Complete. And through experiments, it is found that the proposed algorithm has obvious advantages compared with the previous algorithm. The research of workflow mapping in this paper is divided into the following parts: the presentation of the problem and the proof of complexity. Based on workflow mapping, a mathematical model for end-to-end delay minimization problem with two different reliability requirements is established. Through complexity analysis, it is proved that these two problems are subproblems of a class of NP-Complete problems. It is concluded that these two problems are also NP-Complete. The analysis of the problem and the design of the algorithm. In view of these two problems, the heuristic algorithms RMCWM and EMCWM are proposed, which are divided into virtual machine selection. Because the 3GG model is used in the reliability modeling, based on the analysis of the characteristics of the model, the law of optimizing reliability is found, and the law is integrated into the design of the algorithm. Good results have been achieved. Simulation experiments and performance evaluation. In this paper, the algorithm is designed for different size of the cloud and different number of cloud combinations, The experimental results show that the proposed algorithm is superior to the previous algorithm in the case of small scale cloud combinations and large scale cloud combinations.
【学位授予单位】：西北大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.09;TP311.13

【相似文献】