基于心跳超时机制的Hadoop实时容错技术

发布时间：2018-04-26 03:36

本文选题：云计算 + 心跳机制　；参考：《计算机应用》2015年10期

【摘要】：针对官方的Hadoop软件中提供的节点心跳超时容错机制对短作业并不合理,而且忽略了异构集群中各节点超期时间设置的公平性的问题,提出了公平心跳超时容错机制。首先根据每个节点的可靠性及计算性能构建节点故障误判损失模型,提出公平误判损失(FMJL)算法,使其同时满足长作业和短作业要求;接着,设计并实现了基于FMJL算法的公平超时机制。在实现了公平超时机制的Hadoop上运行大约345 s的短作业时,当出现Task Tracker节点故障时作业完成时间平均大约节省了44%,与自适应超时机制相比,作业完成时间大约节省了23%。实验结果表明,公平超时机制在保证不影响长作业完成时间的情况下缩短了短作业的容错处理时间,提高了Hadoop的实时处理效率。
[Abstract]:The node heartbeat fault tolerance mechanism provided in the official Hadoop software is not reasonable for short operation, and ignores the fairness of each node in the heterogeneous cluster, and puts forward a fair heartbeat timeout fault-tolerant mechanism. First, the node fault misjudgement loss model is constructed according to the reliability and computing performance of each node. A fair misjudgement loss (FMJL) algorithm is proposed to meet both long and short job requirements; then, a fair timeout mechanism based on FMJL algorithm is designed and implemented. When a short operation of about 345 s is run on a fair timeout mechanism, the job completion time is roughly reduced when a Task Tracker node fails. 44%, compared with the adaptive timeout mechanism, the job completion time saves about 23%. experimental results, which shows that the fair timeout mechanism reduces the fault-tolerant processing time of short jobs and improves the real-time processing efficiency of Hadoop without affecting the long job completion time.

【作者单位】：西南交通大学信息科学与技术学院;计算机软件新技术国家重点实验室(南京大学);
【基金】：国家自然科学基金资助项目(61202043,61170111) 网络智能信息处理四川省高校重点实验室开放课题资助项目(SZJJ2014-049)
【分类号】：TP302.8

【参考文献】