一种集群监控容错系统的设计与实现

发布时间：2018-06-23 20:27

本文选题：集群 + 监控　；参考：《电子科技大学》2012年硕士论文

【摘要】：目前，集群技术是当今高性能计算系统的一个重要研究方向，集群系统以其高性价比、高扩展性、高可用性的特点，迅速发展成为高性能计算的一个重要解决方案，在石油物探行业得到广泛应用。本文所研究的是面向地震数据处理的应用集群系统，，随着该集群中节点的数目和提交运行的作业数目越来越多，集群系统的监控和管理成了一个重大问题。伴随集群系统规模的不断扩大，系统发生故障的概率也随之增长，加上地震数据处理作业往往数据量都很大，作业执行时间也很长，如果发生故障造成作业运行失败，将会浪费大量的系统计算资源和时间资源。因此研究该应用集群系统的容错技术具有十分重要的意义。本文研究设计的集群监控容错系统是该地震数据处理应用集群系统的重要支持系统，是专门针对地震数据处理特点开发的特殊辅助系统。针对上文提出的相关问题，本论文的主要工作包括：第一，研究了现有的集群监控系统，设计并实现了面向地震数据处理平台集群系统的监控模块，完成了对整个应用集群系统的监控信息的采集、汇聚和显示等应用的具体实现。监控模块主要包括对集群节点信息和作业信息的监控，为系统管理员和用户进行集群管理和监控提供了方便。第二，设计并实现了利用心跳包技术进行节点故障检测的集群系统容错功能，并且针对该地震数据处理的具体应用完成了节点故障后续处理的相关工作。该集群节点故障检测和处理的功能为本文设计的利用应用级作业检查点实现作业故障容错功能提供了基础。第三，在研究现有检查点技术的基础上，并且针对地震数据处理作业的特点和地震数据处理流程的特殊性，设计并实现了基于地震数据单元的应用级作业检查点设置与回卷恢复功能，该功能结合节点容错功能，能够实现集群作业出现故障后的自动容错。通过实验测试验证了该应用级作业检查点的可行性，提高了集群系统的可用性，能够使作业在遇到故障时从检查点开始继续执行作业，减少了作业重复执行时间，避免了系统大量的计算资源和时间的浪费。
[Abstract]:At present, cluster technology is an important research direction of high performance computing system. With its high performance and price ratio, high scalability and high availability, cluster system has rapidly developed into an important solution for high performance computing. It is widely used in petroleum geophysical exploration industry. In this paper, an application cluster system for seismic data processing is studied. With the increasing number of nodes and jobs submitted to run in the cluster, the monitoring and management of cluster system becomes an important problem. With the continuous expansion of cluster system scale, the probability of system failure also increases, plus seismic data processing jobs are often very large data, job execution time is also very long, if failure caused by job failure, Will waste a lot of system computing resources and time resources. Therefore, it is of great significance to study the fault-tolerant technology of this application cluster system. The cluster monitoring and fault tolerant system studied in this paper is an important support system for the seismic data processing application cluster system. It is a special auxiliary system specially developed for the characteristics of seismic data processing. The main work of this paper is as follows: firstly, the existing cluster monitoring system is studied, and the monitoring module for the cluster system of seismic data processing platform is designed and implemented. The collection, aggregation and display of the monitoring information of the whole application cluster system are realized. Monitoring module mainly includes monitoring of cluster node information and job information, which provides convenience for system administrator and user to manage and monitor cluster. Secondly, the fault-tolerant function of cluster system using heartbeat packet technology to detect node fault is designed and implemented, and the related work of node fault follow-up processing is completed for the specific application of the seismic data processing. The function of the cluster node fault detection and processing provides the foundation for the application level job checkpoint to realize the job fault tolerance function designed in this paper. Thirdly, on the basis of studying the existing checkpoint technology, and aiming at the characteristics of seismic data processing and the particularity of seismic data processing process, An application level job checkpoint setting and rollback recovery function based on seismic data unit is designed and implemented. Combined with node fault-tolerant function, the automatic fault tolerance of cluster jobs is realized. The feasibility of the application level job checkpoint is verified by the experimental test, and the availability of the cluster system is improved, and the job can continue to execute from the checkpoint in the event of failure, thus reducing the time of repeated execution of the job. The waste of computing resources and time in the system is avoided.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP338;TP302.8

【相似文献】