集群环境下的文件调度优化研究

发布时间：2018-03-31 08:53

本文选题：集群管理　切入点：时间片　出处：《太原理工大学》2012年硕士论文

【摘要】：集群就是一组计算机或者工作站以特定的方式连结起来,它们作为一个整体向用户提供一组网络资源和计算资源。通过集群系统,可以获得接近于超级并行计算机的计算能力和处理能力。随着分布式技术和并行计算的快速发展,用户对集群系统的需求日益增长。集群系统提供强大计算能力的同时,面对结构复杂、节点繁多的实际状况,其系统管理是一个值得深入研究的领域。本文研究的主要内容为日本富士通公司的超级集群系统设计实现的一个作业文件传输调度器。调度器主要完成作业运行所需文件传输调度优化任务,以保证系统稳定、资源负载平衡、作业按时执行的情况下,且使得作业依赖文件传输的时间最短,为作业高效执行提供强大的支撑。论文通过对当前集群环境下文件传输方案的研究和分析,并结合实际的集群架构,提出了文件分时间段传输优化算法,并设计实现了集群系统模拟实验。模拟系统包含三大模块,即主控制模块、通信接口模块和文件传输调度器模块。在主控制模块中,实现了客户端管理、节点的管理、用户提交作业的管理以及对集群系统中作业的监控和调度功能；在接口模块中,主要实现了系统控制、共享内存管理以及实现两个模块之间的数据通信功能；在文件分段传输调度模块中,实现了作业依赖文件分段传输的管理和调度。调度算法在采用经典调度算法的同时,实时地根据作业请求信息,引入时间片机制,通过各个节点之间占用时间片的时间值的比较,得出可以最早时间内完成传输任务的调度方案。另外,通过编程还实现了进程间的数据通信,利用互斥锁机制保障了进程间数据同步与安全性。最后通过一些典型的用例测试,表明计算方法正确,调度结果符合预期目标,实现了对传输时间最短的优化目标,为作业高效执行提供了良好的决策支持。
[Abstract]:A cluster is a group of computers or workstations that are connected in a specific way, providing users with a set of network and computing resources as a whole. With the rapid development of distributed technology and parallel computing, users' demand for cluster system is increasing day by day. Cluster system provides powerful computing power at the same time. In the face of complicated structure and numerous nodes, its system management is a field worthy of further study. The main content of this paper is a job file transfer scheduler designed and implemented by Fujitsu Corporation of Japan. The scheduler mainly completes the optimization task of file transfer scheduling needed for job operation, in order to ensure the stability of the system. Resource load balance, job execution on time, and the shortest time for the job to rely on file transfer, provide a strong support for the efficient execution of the job. Through the research and analysis of the file transfer scheme in the current cluster environment, and combining with the actual cluster architecture, the paper puts forward the optimization algorithm of file transmission in different time periods. The simulation system consists of three modules: the main control module, the communication interface module and the file transfer scheduler module. In the main control module, the client management and node management are realized. In the interface module, the functions of system control, shared memory management and data communication between the two modules are realized. In the file segment transfer scheduling module, the management and scheduling of the job dependent file segment transmission are realized. The scheduling algorithm adopts the classical scheduling algorithm and introduces the time slice mechanism according to the job request information in real time. Through the comparison of the time values of each node occupying the time slice, the scheduling scheme which can complete the transmission task in the earliest time is obtained. In addition, the inter-process data communication is realized by programming. The mutual exclusion mechanism is used to guarantee the synchronization and security of inter-process data. Finally, some typical use case tests show that the calculation method is correct, the scheduling result is in line with the expected goal, and the optimal target of the shortest transmission time is achieved. It provides good decision support for job efficient execution.
【学位授予单位】：太原理工大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP338

【参考文献】