负载均衡的Hadoop平台调度算法研究

发布时间：2018-07-28 14:43

【摘要】：近年来,随着计算机信息技术的不断发展,数据信息在各个行业已经呈爆炸式增长。据全球互联网中心数据,截止到2020年底,全球数据量预计将达到35ZB。就像人们说的,大数据时代正式到来了。在处理海量数据的关键时刻,云计算技术随之产生,随着技术不断发展,处理大数据的云计算技术已被广泛应用于公司和企业,例如Google、Microsoft、阿里巴巴,这些公司的云计算系统大多基于一个分布式处理软件框架,即hadoop平台。Hadoop是一个安全、可靠、可并行化运行的开放式源代码框架,并且可实现对应用开发者的透明处理,用户并不需要了解hadoop底层实现细节。但毕竟云计算技术研究时间较短,技术研究还不太深入,云计算仍存在一些挑战,例如安全性,性能,资源管理,可靠性等。资源管理问题与任务调度有关,云计算任务调度是指将用户任务分配给可用资源,以改善任务的执行,提高资源利用率。因此通过改进Hadoop平台任务调度方法可以提高Hadoop平台资源利用率和整体性能。作业调度技术是hadoop平台的核心技术,它主要负责将系统中空闲资源分配给各个作业,以及控制作业执行的顺序,它对于hadoop平台计算资源分配及整体性能起着至关重要的作用。因此我们对于作业调度算法的研究具有重要意义。云计算资源的分配是基于SLA服务的,任务执行成本是任务调度算法的主要性能参数之一。另外,任务调度算法被认为是一个复杂的过程,因为它必须充分利用可用资源来执行大量任务。这就需要开发任务调度算法时应考虑许多参数,从用户的角度来看任务编译时间,成本和响应时间非常重要。而从云提供商的角度来看,资源利用率,容错能力和功耗则很重要。本文首先介绍了hadoop平台的背景、结构及hadoop核心技术;然后对单队列调度(FIFO)、容量调度(Capacity)、公平调度(Fair)三种现有的调度算法作了详尽介绍,并在此基础上对如何改进算法的不足、提高系统整体性能作了介绍,最后提出基于遗传算法(GA)的改进任务调度策略用于分配和执行应用程序的任务,以减少任务完成时间,降低执行成本,并最大限度地提高资源利用率。提出的算法的性能已经使用Cloud Sim工具包进行了评估。
[Abstract]:In recent years, with the continuous development of computer information technology, data information has been explosive growth in various industries. According to the Global Internet Center, by the end of 2020, the global data volume is expected to reach 35 ZB. As they say, the big data era is officially coming. With the development of cloud computing technology, cloud computing technology to deal with big data has been widely used in companies and enterprises, such as Google Microsoft, Alibaba, Most of the cloud computing systems of these companies are based on a distributed processing software framework, hadoop platform. Hadoop is a secure, reliable, parallelized open source code framework, and can achieve transparent processing to application developers. Users do not need to know the details of hadoop's underlying implementation. But after all, cloud computing technology research time is short, technology research is not very deep, cloud computing still has some challenges, such as security, performance, resource management, reliability and so on. Resource management is related to task scheduling. Cloud computing task scheduling is to assign user tasks to available resources to improve task execution and resource utilization. Therefore, the resource utilization and overall performance of Hadoop platform can be improved by improving the task scheduling method of Hadoop platform. Job scheduling is the core technology of hadoop platform. It is mainly responsible for allocating free resources to each job in the system and controlling the order of job execution. It plays an important role in computing resource allocation and overall performance of hadoop platform. Therefore, it is of great significance to study the job scheduling algorithm. The allocation of cloud computing resources is based on SLA services, and task execution cost is one of the main performance parameters of task scheduling algorithm. In addition, task scheduling algorithm is considered to be a complex process, because it must make full use of available resources to perform a large number of tasks. Many parameters should be taken into account when developing task scheduling algorithms. From the point of view of the user, the compilation time, cost and response time of the task are very important. From the point of view of cloud provider, resource utilization, fault tolerance and power consumption are very important. This paper first introduces the background, structure and hadoop core technology of hadoop platform, then introduces the three existing scheduling algorithms of (FIFO), capacity scheduling (Capacity), fair scheduling (Fair) in detail, and on the basis of this, gives a detailed description of how to improve the algorithm. Finally, an improved task scheduling strategy based on genetic algorithm (GA) is proposed to allocate and execute the tasks of the application program, in order to reduce the task completion time and the execution cost. And to maximize resource utilization. The performance of the proposed algorithm has been evaluated using the Cloud Sim toolkit.
【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP301.6

【相似文献】