Hadoop平台下的负载均衡优化研究与改进

发布时间：2018-02-10 02:28

本文关键词： 负载均衡 Hadoop集群分区策略蚁群算法蜂群算法融合算法　出处：《河北经贸大学》2017年硕士论文　论文类型：学位论文

【摘要】：在云计算、大数据环境下,负载均衡问题逐渐成为研究的焦点之一。负载均衡是实现集群最优调度的主要目标之一,计算节点的负载不均衡,就会导致云平台上任务执行效率低、严重浪费资源等问题。当Hadoop集群中任务规模很大并且较多节点负载较高时,进一步优化调度算法可有效避免集群节点间出现负载极其不均衡的情况。本文研究了Hadoop集群的负载均衡机制,并且对分区算法和智能算法分别进行了相应的改进,以提升集群的效率和性能。本文的主要内容包括:(1)基于改进分区策略的动态负载均衡算法研究针对Hadoop平台自带的分区算法未考虑数据值的密集程度而造成的数据非均匀划分情况,本文提出对分区数目进行扩充,并在运行中加入动态任务转移机制,充分利用空闲节点平衡高负载节点,以在保证负载均衡的同时,提高集群的资源利用率。(2)基于双群融合智能算法的负载均衡优化研究充分利用两个不同智能算法各自的优点,克服两者的缺点,可以有效地提高现有算法的优化效果。因此本文利用蚁群算法优秀的全局搜索能力与蜂群算法优秀的横向搜索能力,将两个智能算法进行融合,提出了双群融合智能算法,使二者充分发挥各自的优势,平衡集群的负载,提高集群资源的利用率,提升算法收敛效率,缩短任务执行时间。最后搭建实验环境Hadoop集群,使上述两个算法分别在此集群环境下实现,并进行多次实验对比了改进前与改进后的算法性能,两个改进算法均能有效平衡集群负载,提高集群的资源利用率,缩短作业的执行时间。
[Abstract]:In cloud computing and big data environment, load balancing problem has gradually become one of the focus of research. Load balancing is one of the main objectives to realize the optimal scheduling of cluster, and the load balance of computing nodes is not balanced. It will lead to low efficiency of task execution on cloud platform, serious waste of resources and so on. When the task size in Hadoop cluster is very large and the load of more nodes is high, Further optimization scheduling algorithm can effectively avoid the extremely unbalanced load between cluster nodes. In this paper, the load balancing mechanism of Hadoop cluster is studied, and the partition algorithm and intelligent algorithm are improved respectively. In order to improve the efficiency and performance of the cluster. The main contents of this paper include: 1) dynamic load balancing algorithm based on improved partitioning strategy; data caused by partitioning algorithm based on Hadoop platform without considering the density of data values. Uneven division, In this paper, the number of partitions is expanded, and dynamic task transfer mechanism is added in the operation, which makes full use of idle nodes to balance the high load nodes, so as to ensure load balance at the same time. Research on load balancing Optimization based on dual swarm fusion intelligent algorithm; make full use of the advantages of two different intelligent algorithms to overcome the shortcomings of the two. Therefore, by using the excellent global search ability of ant colony algorithm and the excellent horizontal search ability of bee colony algorithm, the two intelligent algorithms are fused, and a dual colony fusion intelligent algorithm is proposed. So that they can give full play to their respective advantages, balance the load of the cluster, improve the utilization of cluster resources, improve the convergence efficiency of the algorithm, and shorten the task execution time. Finally, the experimental environment Hadoop cluster is built. The above two algorithms are implemented in this cluster environment respectively, and the performance of the improved algorithm before and after the improvement is compared through many experiments. The two improved algorithms can effectively balance the load of the cluster and improve the resource utilization ratio of the cluster. Shorten the execution time of the job.
【学位授予单位】：河北经贸大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;TP18

【相似文献】