云环境下分布式任务调度算法的研究与实现

发布时间：2019-03-16 16:19

【摘要】：云计算是IT领域的一次新的重大变革,推动了新的产业和价值链的发展。云计算平台将大量的计算、存储和网络等资源,统一管理起来,并以网络服务的方式提供给用户,实现了资源协同共享,提高了资源的利用率。但是,由于云资源的规模庞大、异构分布以及动态变化等特点,如何合理有效调度任务,实现合理的资源分配面临着很多的问题。针对云环境中工作流任务调度时资源节点的跨地域异构性特点和执行效率问题,以及数据处理类任务的调度问题,本文做出如下内容的研究:(1)提出了云环境下基于人工免疫算法的工作流调度模型。云环境中资源的区域性分布对工作流任务调度过程中的任务节点间的通信传输产生很大的影响,本文针对这一问题,将工作流任务节点的完成时间和跨节点传输时延作为约束函数,利用人工免疫算法的优势,由约束函数生成资源和任务间的适应度函数,并改进人工免疫算法的抗体变异过程,采用基因重组方法进行抗体的定向变异,提出一种基于人工免疫算法的工作流调度模型,以得到工作流任务节点和资源节点之间的合理调度方案。并通过实验验证了该算法在资源跨区域分布的工作流调度中可提高任务的执行效率。(2)提出了云环境下基于数据分片的任务调度策略。针对数据处理类任务的调度过程,需要使用合理的调度方案将数据分片处理,并进行征用资源,将数据分片分配到征用的资源节点上进行数据处理,本文提出一种基于数据分片的任务调度策略,可依据数据的可切分粒度,以及可用资源节点的性能差异,建立数据分片的优化调度数学模型,求解出数据分片的理想切分比例,再结合数据的实际可切分粒度,经过二次分片可得到任务的调度策略。通过实验验证了采用该策略可有效缩短数据处理类任务的完成时间。(3)在基础设施管理平台的基础上,设计并开发了数据服务云平台,并在该平台中基于本文提出的任务调度算法,实现了调度管理模块。本文对调度管理模块的计算中心、征用中心、数据中心及征用机服务做出了详细的设计描述。通过系统测试,展示了本文提出的调度算法在资源征用场景中的效果。
[Abstract]:Cloud computing is a new major change in the field of IT, promoting the development of new industries and value chains. Cloud computing platform manages a large number of resources, such as computing, storage and network, and provides it to users in the way of network service. It realizes the cooperative sharing of resources and improves the utilization of resources. However, due to the large scale of cloud resources, heterogeneous distribution and dynamic changes, how to reasonably and effectively schedule tasks and achieve reasonable resource allocation is facing a lot of problems. In view of the cross-geographical heterogeneity and execution efficiency of resource nodes in workflow task scheduling in cloud environments, and the scheduling of data processing tasks, The main contents of this paper are as follows: (1) A workflow scheduling model based on artificial immune algorithm in cloud environment is proposed. The regional distribution of resources in cloud environment has a great influence on the communication transmission between task nodes in workflow task scheduling process. This paper aims at this problem. The completion time and cross-node transmission delay of workflow task nodes are regarded as constraint functions. Taking advantage of the advantage of artificial immune algorithm, the fitness function between resources and tasks is generated from the constraint function, and the antibody mutation process of artificial immune algorithm is improved. This paper presents a workflow scheduling model based on artificial immune algorithm to obtain a reasonable scheduling scheme between task nodes and resource nodes by means of gene recombination for directed variation of antibodies. Experiments show that the proposed algorithm can improve the efficiency of task execution in workflow scheduling with cross-regional distribution of resources. (2) A data slicing-based task scheduling strategy is proposed in the cloud environment. In view of the scheduling process of data processing tasks, it is necessary to use a reasonable scheduling scheme to segment the data and requisition the resources, and allocate the data fragments to the requisitioned resource nodes for data processing. In this paper, a task scheduling strategy based on data slicing is proposed. According to the granularity of data and the performance difference of available resource nodes, a mathematical model for optimal scheduling of data slicing is established, and the ideal slicing ratio of data is solved. Combined with the actual granularity of the data, the scheduling strategy of the task can be obtained through the secondary slicing. Experiments show that this strategy can effectively shorten the completion time of data processing tasks. (3) on the basis of infrastructure management platform, a data service cloud platform is designed and developed. In this platform, based on the task scheduling algorithm proposed in this paper, the scheduling management module is implemented. In this paper, the computing center, requisition center, data center and enlistment service of the dispatching management module are described in detail. Through the system test, it shows the effect of the proposed scheduling algorithm in the resource requisition scenario.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP3;TP18

【参考文献】