云计算环境下资源调度策略的研究与实现

发布时间：2018-05-23 16:59

本文选题：云计算 + Hadoop　；参考：《北京邮电大学》2014年硕士论文

【摘要】：如今我们已经步入了一个全新的时代——大数据时代。我们每天的生活、学习和工作都要产生上T级别的网络数据量。因此,随着互联网技术的飞速发展而产生的一种新型的分布式计算模式云计算应运而生。它的出现解决了海量数据存储和实时备份等一系列和大数据息息相关的问题。近年来,国内外各大互联网公司和学者专家们都积极投身云计算技术的研究,资源调度则是云计算研究的关键技术之一。云计算环境中资源调度策略的优劣直接影响整个集群系统的性能。因此对云计算环境中资源调度策略的研究具有十分重大的意义。本文主要的研究内容如下所示： (1)研究分析了云计算环境下的资源调度策略。分析了Hadoop平台的系统结构、MapReduce框架和分布式文件系统HDFS。 (2)深入研究了开源分布式Hadoop平台的资源调度策略。并对Hadoop平台常用的三种资源调度策略：FIFO、计算能力和公平分配资源调度策略进行了仿真实验。通过实验结果来对比分析三种资源调度策略在系统吞吐率和系统响应时间的性能区别。 (3)针对Hadoop平台常用资源调度策略没有考虑作业任务有时间限制的问题,本文提出了一种基于LLF的Deadline资源调度策略——LLFD。设计了数据本地性管理算法、Master的全局调度算法和系统节点本地调度算法。 (4)在CloudSim仿真平台上验证LLFD资源调度策略的性能。通过实验仿真结果可以证实,该资源调度策略可以在云计算大规模的基础设施环境中尽可能使每个作业任务在截止日期之前得到执行,同时能够使集群中满足时间期限的节点数目达到最小化,在可接受的超时时间范围内实现了最小化由于超时带来的惩罚率。从而可以提升整个集群系统的性能。
[Abstract]:Now we have entered a new era-big data era. Our daily life, study and work must produce T-level network data. Therefore, with the rapid development of Internet technology, a new distributed computing model cloud computing came into being. It solves a series of problems related to big data, such as massive data storage and real-time backup. In recent years, domestic and foreign major Internet companies and scholars and experts are actively engaged in the research of cloud computing technology, resource scheduling is one of the key technologies in cloud computing research. The resource scheduling strategy in cloud computing environment directly affects the performance of the whole cluster system. Therefore, the research of resource scheduling strategy in cloud computing environment is of great significance. The main contents of this paper are as follows: 1) the resource scheduling strategy in cloud computing environment is studied and analyzed. The system structure of Hadoop platform and the distributed file system HDFS are analyzed. The resource scheduling strategy of open source distributed Hadoop platform is studied in detail. Three kinds of resource scheduling strategies: FIFO, computational power and fair resource allocation are simulated on Hadoop platform. The performance differences of the three resource scheduling strategies in the throughput and response time of the system are compared and analyzed by the experimental results. 3) aiming at the problem that the resource scheduling policy in common use in Hadoop platform does not consider the time limit of job tasks, this paper proposes a Deadline resource scheduling strategy based on LLF, LLFD-LLFD. The global scheduling algorithm of Master and the local scheduling algorithm of system node are designed. 4) verify the performance of LLFD resource scheduling strategy on CloudSim simulation platform. The simulation results show that the resource scheduling strategy can make every task execute before the deadline in the large-scale infrastructure environment of cloud computing. At the same time, it can minimize the number of nodes meeting the time limit in the cluster, and minimize the penalty rate due to the timeout in the acceptable timeout range. Thus, the performance of the whole cluster system can be improved.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.01

【参考文献】