基于支持向量机的HDFS副本放置改进策略

发布时间：2018-05-10 06:03

  本文选题：支持向量机 + 云存储　； 参考：《计算机工程》2015年11期

【摘要】：为实现超大规模数据的存储并提高容错性,Hadoop分布式文件系统(HDFS)采用一种机架感知的多副本放置策略。但在放置过程中没有综合考虑各节点服务器的差异性,导致集群出现负载失衡。由于放置时采用随机方式,造成节点之间的网络距离过长,使得传输数据会消耗大量时间。针对以上问题,提出一种基于SVM的副本放置策略。通过综合考虑节点负载情况、节点硬件性能、节点网络距离为副本找到最佳的放置节点。实验结果表明,与HDFS原有的副本放置策略相比,该策略能更有效地实现负载均衡。
[Abstract]:In order to store large scale data and improve fault tolerance, Hadoop distributed file system (HDFS) adopts a rack aware multi-replica placement strategy. However, in the process of placement, the differences of node servers are not considered synthetically, which leads to the load imbalance of the cluster. The network distance between nodes is too long because of the random way of placement, which makes the transmission of data consume a lot of time. Aiming at the above problems, a replica placement strategy based on SVM is proposed. By considering the load of the node, the hardware performance of the node and the distance of the node network to the replica, the optimal placement node is found. The experimental results show that the proposed strategy is more effective than the original replica placement strategy of HDFS.
【作者单位】： 重庆大学计算机学院;
【分类号】：TP333;TP18
，

本文编号：1868100