大规模云数据中心负载优化调度方法研究

发布时间：2018-02-28 03:35

本文关键词： 云数据中心突发性负载自相似性负载负载生成负载监控负载优化调度　出处：《浙江大学》2014年博士论文　论文类型：学位论文

【摘要】：负载优化调度是指对负载进行合理高效地调度处理以实现特定的优化目标。随着大规模云数据中心在全球范围内的广泛部署,云数据中心负载优化调度正成为关系其服务能力建设及运营成本的重要方面,也是当前学术界的一个研究热点。而传统的负载优化调度方法不能很好地适应云计算应用的新特性,也无法很好地解决云计算带来的规模和复杂度挑战。本文围绕大规模云数据中心负载优化调度框架,云数据中心Web负载、Job负载及云数据中心之问批数据传输负载的优化调度方法展开研究工作,取得以下成果：针对传统负载优化调度系统可扩展性差,对云应用负载特性及智能化调度决策支持力度有限等不足,为大规模云数据中心提出一种可扩展、高效、智能化的负载优化调度系统JTangWOS. JTangWOS由基于数据分发服务DDS (Data Distribution Service)的负载监控平台,基于复杂事件处理CEP (Complex Event Processing)的调度决策支持平台,以及具体的负载优化调度系统组成。它能够很好地适应云数据中心的规模挑战,支持海量的监控数据收集和传输,支持各种复杂负载管理与调度决策的制定,并具有良好的可扩展性。实验还验证了该系统在负载优化调度方面的高效性(并发性和吞吐量分别提升4.8倍和20倍)及智能性。针对云计算应用基准程序Cloudstone,提出一种基于两阶段马尔科夫调制泊松过程MMPP2 (2-state Markov Modulated Possion Process)叠加模型的突发性和自相似性Web负载联合生成方法BURSE.该方法可以解决当前突发性和自相似性负载分别生成方法不能很好满足真实云Web应用负载情况的不足。与传统方法相比,该方法生成的负载更符合实际情况,并且所用方法更直观。接着,论文为大规模云数据中心系统提出了一种可感知突发性和自相似性的自适应负载均衡方法。实验表明所提负载生成方法具有很高的精度(各种突发性和自相似性强度组合平均误差都小于10%)及鲁棒性(精度不随负载样本数的增加而降低)。与传统贪婪和完全随机负载均衡算法相比,所提可感知突发性和自相似性的大规模云数据中心自适应负载均衡方法具有更好的性能和效率。针对大规模云数据中心地理位置分布的差异性,提出一种基于交替方向多乘子方法ADMM (Alternating Direction Method of Multipliers)的分布式算法,用来解决带有异构资源需求的Job在多个带有异构服务器的数据中心之间的优化调度问题。通过考虑不同数据中心所在地区之间的电价差异及用户请求到不同数据中心的网络延迟,为异构云数据中心多Job负载实现了一种成本最低的联合优化调度方法。实验表明,该方法在经过较少的几十次迭代(最多60次,80%的时间只需33次)后会收敛于一个实际中可接受的近似最优解。对比其他没有考虑数据中心和服务器联合调度的方法,所提方法能够完全保证用户服务质量QoS(Quality of Service)不被违反。对比其他只优化能耗成本或网络延迟效用损失的联合调度方法,所提方法能在效用损失和能耗成本之间实现一个更好的均衡,具有最低的总成本。针对云数据中心之间的批数据传输(Inter-DC Bulk Data Transfer)负载,通过利用批数据传输较高的延迟容忍性及不同地区数据中心电价的差异性,提出了一种能耗成本最低的两阶段优化调度方法。论文系统地研究了在大规模云数据中心网络中如何路由及调度批数据传输任务使得总能耗成本最小的问题,并用最小成本多并发流对该问题进行了建模。同时,还充分利用云数据中心之间批数据传输在时间和空间上的灵活性,提出一种高效的两阶段优化调度方法来求解该模型。基于真实云数据中心网络拓扑及真实历史电价数据,实验表明所提方法在降低数据中心能耗成本方面优于当前已有各种批数据传输优化调度方法及策略,其能耗成本降低率在闲时(凌晨)和忙时(傍晚)分别达到了31%和48%。另外,实验还表明所提方法具有较低的时间复杂度(与给定的可传输时间近似成线性关系),能够在性能和时间复杂度上实现一个好的均衡。
[Abstract]:Load scheduling refers to the load reasonably and efficiently processing to achieve specific optimization objectives. With the large scale cloud data centers are widely deployed in the global scope, cloud data center load scheduling is becoming the important aspect of the service ability construction and operation cost, it is also a research focus in the academic circles. But the traditional load optimal dispatch method can't adapt to the new features of cloud computing applications, is not a good solution for cloud computing brings the scale and complexity of the challenges. This paper focuses on large scale cloud data center load optimization scheduling framework, cloud data center Web load, Job load and cloud data center asked the optimization scheduling method batch data transmission load to launch the research work, has achieved the following results: Based on the traditional load optimal dispatching system of poor scalability of cloud application load characteristic and intelligent adjustment The limited degree of decision support, for the large-scale cloud data center provides a scalable, efficient, intelligent load optimal dispatching system of JTangWOS. JTangWOS by DDS based on data distribution service (Data Distribution Service) load monitoring platform, complex event processing based on CEP (Complex Event Processing) scheduling decision support platform, and the concrete load scheduling system. It can well adapt to the cloud data center scale challenges, support the massive monitoring data collection and transmission, support a variety of complex load management and decision making, and has good scalability. The experiment also validated the system load in optimal scheduling efficiency (the concurrency and throughput respectively increased 4.8 times and 20 times) and intelligence. According to the application of cloud computing benchmark program Cloudstone, based on the two phase of Marco Schiff (2-state MMPP2 modulated Poisson process Markov Modulated Possion Process) model burstiness and self similarity of Web combined with BURSE. load generation method this method can solve the burstiness and self similarity load respectively generation method can not satisfy the real application problems cloud Web load conditions. Compared with the traditional method, the method of load the more in line with the actual situation, and the method is more intuitive. Then, this paper puts forward a perceived bursty and self similar adaptive load balancing method of large-scale cloud data center system. Experiments show that the proposed load generation method with high accuracy (sudden and self similarity of intensity the average error is less than 10%) and robustness (accuracy and reduce with increase of load sample number). Compared with the traditional greedy and completely random load balancing algorithm, the proposed. Know the bursty and self similar large-scale cloud data center adaptive load balancing algorithm has better performance and efficiency. According to the difference of large-scale cloud data center location distribution, we proposed an alternating direction multiplier method based on ADMM (Alternating Direction Method of Multipliers) distributed algorithm is used to solve scheduling problem with heterogeneous the Job resource requirements among multiple heterogeneous servers with the data center. The price difference between considering different data center area and user requests to different data center network latency for heterogeneous cloud data center to achieve a joint optimization scheduling method of the lowest cost load Job. Experimental results show that this method in after several iterations less (up to 60 times, 80% time only 33) will converge to a practical approximation acceptable Optimal solution. Compared with other methods without considering the joint operation of the data center and server, the proposed method can fully guarantee the quality of service QoS (Quality of Service) is not violated. Compared to the other only optimize energy utility loss cost or network delay scheduling method, the proposed method can achieve a better balance between the utility the loss of energy consumption and cost, with the lowest total cost. According to the number of data transmission between the cloud data center (Inter-DC Bulk Data Transfer) load, through delay difference tolerance data center and different areas of electricity using batch data transmission is high, a two stage energy consumption of the lowest cost optimal scheduling method for the system. Study on how to route and schedule batch data transmission task makes the problem of minimum cost in the total energy consumption of large-scale cloud data center network, and with minimum cost Multi flow on the modeling of the problem. At the same time, but also make full use of the flexibility in time and space on the number of data transmission between the cloud data center, this paper presents a new two stage optimization scheduling method to solve the model. The real cloud data center network topology and the true history of price data based on the experiment shows that the proposed method to reduce data center energy cost than the current batch of various data transmission scheduling method and strategy, its cost is to reduce the rate at leisure (Ling Chen) and busy (evening) respectively reached 31% and 48%. in addition, the experiment also shows that the proposed method has low time complexity (and can be given the transmission time is approximately linear), can achieve a good balance between performance and complexity.

【学位授予单位】：浙江大学
【学位级别】：博士
【学位授予年份】：2014
【分类号】：TP308;TP393.01

【相似文献】