分布式计算框架Antnest的任务调度设计与实现

发布时间：2018-06-06 06:34

本文选题：分布式 + 计算框架　；参考：《华中科技大学》2012年硕士论文

【摘要】：近年来随着计算机网络技术的高速发展，互联网企业为用户提供的应用和个性化服务也越来越多，随之而来的是用户提交的海量数据，企业必须及时而又正确地处理这些数据并向用户反馈结果。通常，人们会选择使用分布式计算系统来完成这些复杂的任务。然而传统的分布式计算平台往往需要配置大型的服务器集群，还需要专门的技术人员去跟进和维护，这给很多企业都带来了巨大的经济开销。轻量级分布式任务计算框架Antnest采用了客户机/服务器计算模式和三层体系结构，可以将工作任务分布在廉价的任务节点上，中心节点负责整个系统的调度工作，任务节点采用多JVM多线程执行任务。在Antnest中，提交的任务都是经过定制的，不同的任务对实时性的要求各不相同，针对此，Antnest为任务调度系统设计了一种多级队列抢占式调度算法，该算法充分考虑到任务对实时性的要求，高优先级任务可以抢占系统，被抢占任务会移入高优先级队列中等待调度，同时利用任务的预定启动时间来确保任务执行的有效性。另外，Antnest还设计了动态任务分配算法来完成子任务的分配工作。在动态分配算法中，子任务和任务节点的分配关系是在任务的执行过程中动态确定的，Antnest会优先将子任务分配给最先执行完子任务的任务节点。在任务调度系统的实现中，任务调度系统各模块间通过自定义的通信协议完成消息通信，，通过指定的任务接口触发任务逻辑，利用任务节点的状态管理机制完成资源管理并保证Antnest的可扩展性，同时，利用心跳检测和子任务重传的容错机制确保任务的正确执行。测试结果表明，Antnest系统能有效地提高任务的执行效率，减短执行时间；同时，结果也验证了Antnest系统中多级队列抢占式调度算法的合理性和可行性。
[Abstract]:In recent years, with the rapid development of computer network technology, Internet enterprises provide users with more and more applications and personalized services, followed by massive data submitted by users. The enterprise must process the data in a timely and correct manner and feedback the results to the user. Usually, people choose to use distributed computing systems to accomplish these complex tasks. However, traditional distributed computing platforms often need to configure large server clusters, and also need specialized technicians to follow up and maintain, which brings huge economic expenses to many enterprises. The lightweight distributed task computing framework (Antnest) adopts the client / server computing model and three-tier architecture, which can distribute the work tasks on the cheap task nodes, and the central node is responsible for the scheduling of the whole system. The task node uses multi-JVM multi-thread to execute the task. In Antnest, all the submitted tasks are customized, and different tasks require different real-time performance. A multi-level queue preemptive scheduling algorithm is designed for this task scheduling system. The algorithm fully takes into account the real-time requirements of tasks, high priority tasks can preemptive system, preemptive tasks will be moved into high priority queue to wait for scheduling, and the scheduled start time of tasks will be used to ensure the effectiveness of task execution. In addition, Antnest designs a dynamic task allocation algorithm to accomplish the task assignment. In the dynamic assignment algorithm, the relationship between subtasks and task nodes is dynamically determined in the process of task execution. In the implementation of task scheduling system, message communication is accomplished between modules of task scheduling system through self-defined communication protocol, and task logic is triggered by specified task interface. The state management mechanism of the task node is used to accomplish resource management and to ensure the scalability of the Antnest. At the same time, the fault-tolerant mechanism of heartbeat detection and sub-task retransmission is used to ensure the correct execution of the task. The test results show that the Antnest system can effectively improve the task execution efficiency and reduce the execution time, and it also verifies the rationality and feasibility of the multilevel queue preemptive scheduling algorithm in Antnest system.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP338.8

【参考文献】