智能电网云平台调度策略的研究

发布时间：2018-01-29 21:15

本文关键词： 智能电网云平台 Hadoop 作业调度云计算推测执行任务　出处：《华北电力大学》2014年硕士论文　论文类型：学位论文

【摘要】：随着我国智能电网事业的发展，全国电力系统互联已成为一个趋势，大量的先进的数据采集与监控设备、相量测量单元（PMU）、智能电表等被应用，现代电力系统正在演变成一个集聚大数据和信息的计算系统。针对智能电网对海量的数据存储和大规模并行计算的迫切需求，鉴于电力系统广域网的完整性，学者提出了整合网内现有计算和存储资源，建立电力私有云的概念。Hadoop是主要由HDFS和MapReduce组成的开源云计算项目，可以部署在普通个人计算机上，从而组成廉价的云平台。作业的调度算法对云计算有着至关重要的作用，它是解决作业在什么地点、什么时间执行的问题。智能电网云平台依托于各级电网的计算资源，集群中普遍资源存在着节点异构问题，异构节点的执行能力的不同和用户提交作业任务量不同，会导致比较突出的任务同步问题。根据该情况，本文在hadoop平台下，给出了一种基于作业执行时间预测的资源优化推测执行算法，该算法通过预先执行作业一部分任务，通过这些先行任务预测作业平均和整体的运行时间，同时将群集中的节点以执行相同作业所属任务的执行时间为参数，将节点分为快节点和慢节点，而推测执行的任务只能发生在快节点上，该算法结合任务执行节点的性能参数，判断该任务是否进行推测执行，当推测执行发生时会尽可能以局部执行的方式执行其后备任务，推测任务发生之前，，该算法会检查群集中其它节点执行该任务的成本是否低于该节点（主要以inputsplit的所在节点与执行节点的距离做参考），如果任务在其它节点执行成本更低，则算法会放弃本次推测执行。本文通过实验比较了该算法和、计算能力调度算法、公平调度算法、基于高优先级滑动窗口调度算法的优缺点，通过分别代表内存、CPU、网络等不同类型资源的云计算应用例程WordCount、CPUActivity、URLGet，进行三组，每组六次实验的测试，结果表明该算法在任务的时间消耗上，推测执行的发生率，网络资源的占用率上均有明显的减小，整体上缩短了资源的消耗，并提高了任务的完成速度。因此在一定程度上适合节点众多，拓扑结构复杂，节点差异大的电力系统私有云的作业调度的需求。
[Abstract]:With the development of smart grid in China, the interconnection of national power system has become a trend. A large number of advanced data acquisition and monitoring equipment, phasor measurement unit (PMU), intelligent meter and so on have been applied. Modern power system is evolving into a computing system that gathers big data and information. In view of the urgent demand of smart grid for massive data storage and large-scale parallel computing, considering the integrity of power system wide area network (WAN). Scholars put forward the concept of integrating the existing computing and storage resources in the network and establishing the power private cloud. Hadoop is an open source cloud computing project mainly composed of HDFS and MapReduce. It can be deployed on an ordinary personal computer to form a cheap cloud platform. Job scheduling algorithms play a vital role in cloud computing, which is a solution to where jobs are located. The problem of when to execute. The cloud platform of smart grid depends on the computing resources of all levels of power grid, and the problem of node heterogeneity exists in the common resources in the cluster. The different execution ability of heterogeneous nodes and the number of jobs submitted by users will lead to the problem of task synchronization. According to this situation, this paper presents a resource optimization algorithm based on job execution time prediction based on hadoop platform, which performs a part of the job tasks in advance. At the same time, the nodes in the cluster are divided into fast node and slow node with the execution time of the same job as the parameter. The proposed task can only occur on the fast node, and the algorithm combines the performance parameters of the task execution node to determine whether the task is supposed to be executed. When speculation execution occurs, it performs its backup tasks as locally as possible, prior to the occurrence of the speculated task. The algorithm checks whether the cost of performing this task for other nodes in the cluster is lower than that of the node (mainly referenced by the distance between the node where the inputsplit is located and the node of execution). If the cost of task execution in other nodes is lower, then the algorithm will give up this speculative execution. This paper compares the algorithm with the computational power scheduling algorithm and the fair scheduling algorithm through experiments. Based on the advantages and disadvantages of high priority sliding window scheduling algorithm, cloud computing application routine WordCount, which represents different types of resources, such as memory, network and so on, is adopted. CPUActivityTurlGet3 groups, each group of six experiments, the results show that the algorithm in the task of time consumption, the incidence of execution. The occupancy rate of network resources is obviously reduced, the overall consumption of resources is shortened, and the speed of task completion is improved. Therefore, to some extent, it is suitable for many nodes and complex topology. The demand for job scheduling of private cloud in power system with large node difference.
【学位授予单位】：华北电力大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.09;TM73

【相似文献】