云环境下能量高效的任务调度方法研究与应用

发布时间：2019-02-18 18:39

【摘要】：随着信息化技术的发展和应用,各行业产生的数据呈爆炸式增长。传统数据库技术不能很好地解决海量数据、高并发、快速响应、可扩展性等大数据应用问题。因此如何高效地存储和管理这些数据,是目前亟需解决的问题。云计算的经济性、可扩展性、容错性等特点使其成为大数据管理的支撑技术。随着云计算的广泛应用,数据中心的数量和规模快速增长,电费开销已经超过了硬件设备本身的购置费用,且仍处于持续增长状态。快速增长的能量消耗同时加剧全球能源危机和环境污染。因此研究云环境下能量高效的数据管理技术刻不容缓。云环境下能量高效的任务调度技术是云环境下能量高效的数据管理技术的重要组成部分,目的是通过任务调度的方法降低用于任务处理的节点在任务处理过程中的能量消耗。本文分别从单节点和多节点两个层面研究云环境下的能量高效的任务调度技术。本文的主要创新点如下:(1)由于现有的多核处理器节点上能量高效的任务调度方法通常假设可以独立控制各个内核的状态,在能耗计算阶段只考虑处理器本身的能耗而未考虑节点其他部件的能耗,且对能效的度量方式未考虑实际意义,因此提出一种面向多种内核结构的代价感知的任务调度框架。综合考虑处理器的静态功耗、动态功耗以及节点其他部件功耗,使用经济代价度量节点进行任务处理的时间代价和能量代价,将任务处理时间、等待时间和能量消耗代价统一。在此基础上分别针对独立控制、整体控制和分组控制三种内核结构的处理器设计不同的任务调度算法。使用该调度框架和任务调度算法可以降低节点的任务处理代价:当内核为独立控制结构时,负载越轻该方法相对传统方法的优势越明显;当内核为整体控制结构时,随着负载加重该方法的节点代价低于传统方法且二者之间的差距越来越大;当内核为分组控制结构时,该方法的节点代价与传统方法相比成倍减少。(2)现有的云环境任务层能量高效的数据密集型任务调度方法主要通过改变数据存储策略实现能量高效,此类方法与具体的数据存储方法和存储介质相关,不具备通用性,因此提出一种同构节点环境下与数据存储策略无关的能量高效的数据密集型任务处理方法EABD。在任务调度过程中综合考虑处理任务的节点数和节点间的负载均衡情况,降低任务处理过程中的能量消耗。尽管该方法相对于传统方法使用更多的节点处理任务,但其任务处理过程中的能量消耗比传统方法更小,在某些情况下其能量消耗甚至不足传统方法的50%。该算法的能量消耗受副本数量影响较小,且在默认3个副本的情况下该算法造成的能量浪费最少。(3)针对实际云环境下的节点异构性问题,提出一种异构节点环境下与数据存储策略和存储介质无关的能量高效的数据密集型任务处理方法MinBalance,将任务调度过程分为节点选择和负载均衡两步。在节点选择过程中,定义四种不同的节点权值,根据贪心算法选择权值最小的节点进行任务处理。在负载均衡阶段对参与任务处理的节点的负载进行均衡,减少节点因等待而造成的能量浪费。该方法充分考虑节点的性能和功耗的异构性,降低任务处理的能量消耗,当待处理的数据量较大时,MinBalance可减少约60%的能量消耗。(4)针对目前能量高效的虚拟机调度方法主要考虑任务特征和资源分配,仅通过减少节点使用数量降低能耗的问题,提出一种云环境下能量高效的虚拟机调度算法EEVS。首先将虚拟机分配到拥有足够资源且最优性能功率比最高的物理机上执行,从节点层减少能量消耗。在虚拟机执行过程中采用基于DVFS的单节点节能技术,通过虚拟机迁移进行资源整合,减少每个物理机的能量消耗,从部件层进一步降低系统的能量消耗。EEVS算法在不造成明显效率降低的情况下,可以节约超过10%的能量消耗。(5)针对目前云计算应用中仅考虑方法的可行性和效率提升,而不重视对能耗的优化等问题,研究并分析了数据挖掘中常用的频繁模式挖掘的能耗问题。根据本文提出的能量高效的任务调度方法,设计一种能量高效的任务调度器EEScheduler,对云环境下频繁模式挖掘中的Map任务进行能量高效的调度,从而降低系统的能量消耗。在四个节点组成的云平台上进行频繁模式挖掘,实验结果表明EEScheduler可以降低60%以上的能量消耗。
[Abstract]:With the development and application of the information technology, the data generated in each industry is exploding. The traditional database technology can not solve the large-scale data application problem such as massive data, high concurrency, fast response, and expandability. Therefore, how to efficiently store and manage these data is a problem that needs to be solved at present. The economics, scalability and fault-tolerance of cloud computing make it the supporting technology of large data management. With the wide application of cloud computing, the number and size of the data center are growing rapidly, and the cost of the electricity bill has exceeded the purchase cost of the hardware equipment itself, and is still in the state of continuous growth. The rapid increase of energy consumption also exacerbates the global energy crisis and environmental pollution. So it is urgent to study the energy-efficient data management technology in the cloud environment. The energy efficient task scheduling technology in the cloud environment is an important part of the energy efficient data management technology in the cloud environment, and the aim of the invention is to reduce the energy consumption of the node used for task processing in the task processing process through the method of task scheduling. This paper studies the energy efficient task scheduling technology in the cloud environment from the single node and the multi-node. The main innovation points of this paper are as follows: (1) Since the energy efficient task scheduling method on the existing multi-core processor nodes generally assumes that the states of the respective cores can be controlled independently, only the energy consumption of the processor itself is taken into account in the energy consumption calculation stage, and the energy consumption of other components of the nodes is not taken into account, and therefore, a cost-aware task scheduling framework for a plurality of kernel structures is proposed. and comprehensively considering the static power consumption, the dynamic power consumption of the processor and the power consumption of other components of the node, and the time cost and the energy cost of the task processing are carried out by using the economic cost metric node, and the task processing time, the waiting time and the energy consumption cost are unified. On the basis of this, the different task scheduling algorithms are designed for the processor of the three kernel architectures, such as the independent control, the whole control and the packet control. using the scheduling framework and the task scheduling algorithm, the task processing cost of the nodes can be reduced; when the core is an independent control structure, the more the load is lighter; the more obvious the advantages of the method relative to the traditional method; when the core is an integral control structure, As the load increases the cost of the node of the method is lower than that of the traditional method and the gap between the two is larger and larger; when the core is a packet control structure, the node cost of the method is reduced by a plurality of times compared with the traditional method. and (2) the energy-efficient data-intensive task scheduling method of the existing cloud environment task layer mainly realizes the energy efficiency by changing the data storage strategy, and the method is related to the specific data storage method and the storage medium, does not have the universality, and therefore, an energy-efficient data-intensive task processing method, EABD, which is independent of the data storage strategy under the environment of a homogeneous node is proposed. in the task scheduling process, the node number of the processing task and the load balance among the nodes are comprehensively considered, and the energy consumption in the task processing process is reduced. Although the method uses more node processing tasks with respect to the conventional method, the energy consumption in its task processing is smaller than that of the conventional method, and in some cases its energy consumption is even less than 50% of the conventional method. The energy consumption of the algorithm is less affected by the number of copies, and the energy wasted by this algorithm is minimized in the case of a default of 3 copies. (3) A data-intensive task processing method, MinBalance, which is independent of the data storage strategy and the storage medium under the environment of a heterogeneous node is proposed, and the task scheduling process is divided into two steps: node selection and load balancing. in the node selection process, four different node weight values are defined, and the task processing is carried out according to the node with the smallest selection value of the greedy algorithm. the load balancing stage equalizes the load of the node participating in the task processing, and reduces the energy waste caused by the node waiting. The method fully considers the heterogeneity of the performance and power consumption of the node, reduces the energy consumption of the task processing, and when the amount of data to be processed is large, the MinBalance can reduce the energy consumption of about 60%. and (4) aiming at the problem that the current energy efficient virtual machine scheduling method mainly considers the task characteristics and the resource allocation, and only reduces the energy consumption by reducing the use quantity of the nodes, and proposes an energy efficient virtual machine scheduling algorithm EEVS under the cloud environment. first, the virtual machine is allocated to a physical machine that has sufficient resources and the optimal performance power is higher than the highest, and energy consumption is reduced from the node layer. During the execution of the virtual machine, the single-node energy-saving technology based on the DVFS is adopted, the resource integration is carried out through the migration of the virtual machine, the energy consumption of each physical machine is reduced, and the energy consumption of the system is further reduced from the component layer. The EEVS algorithm can save more than 10% of the energy consumption without causing significant efficiency degradation. (5) For the current cloud computing application, only the feasibility and efficiency of the method are considered, and the problem of energy consumption optimization and the like is not paid attention to, and the energy consumption problem of the frequent pattern mining used in data mining is analyzed and analyzed. According to the energy efficient task scheduling method proposed in this paper, an energy efficient task scheduler (EScheduler) is designed to efficiently and efficiently schedule the Map tasks in frequent pattern mining in the cloud environment, so as to reduce the energy consumption of the system. Frequent pattern mining is carried out on a cloud platform composed of four nodes, and the results show that the EEScheduler can reduce the energy consumption of more than 60%.
【学位授予单位】：南京航空航天大学
【学位级别】：博士
【学位授予年份】：2016
【分类号】：TP301.6

【相似文献】