基于Kafka的消息队列在射天文数据处理中应用的研究

发布时间：2018-01-10 10:07

本文关键词：基于Kafka的消息队列在射天文数据处理中应用的研究　出处：《昆明理工大学》2017年硕士论文　论文类型：学位论文

【摘要】：明安图射电频谱日像仪(MingantU Spectral Radioheliograph,MUSER)每天生成的数据量可达TB级,并且需要实现海量数据的实时和离线处理,如何较好地解决MUSER系统的海量射电天文数据处理问题,是MUSER项目研发中的一项重要课题。为了满足海量数据处理的实时性和并行性需求,海量数据处理系统通常使用消息队列来提高系统的整体数据处理能力。本文以MUSER系统的海量射电天文数据处理为研究背景,分析MUSER系统的海量数据处理需求并深入了解消息队列技术。为了提高MUSER系统的整体数据处理能力,本文所做的主要工作如下:(1)经过消息队列技术选型,选择将Kafka应用于MUSER系统,把海量数据处理解耦为任务发布和任务处理两部分,提高MUSER系统的并行性和实时性,实现了 MUSER系统的多任务并行处理,初步提高了 MUSER系统的整体数据处理能力。(2)MUSER系统需要实现消费者负载均衡来优化任务调度,但原生的Kafka消费者负载均衡算法无法满足MUSER系统的需求。本文设计并实现了一种优化的消费者负载均衡算法,通过选举管理者控制负载均衡来优化消费者负载均衡过程,进一步提高了 MUSER系统的整体数据处理能力,并通过选举监控者进行异常监控来提高MUSER系统的容错性。(3)MUSER系统使用Kafka进行海量数据处理时,消费异常情况会降低射电天文数据处理结果的准确性。本文设计并实现了消费异常处理机制,通过进行双端日志管理、异常处理和偏移量一致性校验来提高MUSER系统数据处理过程的可靠性和容错性,从而提高MUSER系统的海量射电天文数据处理结果的准确性。本文的研究成果提高了 MUSER系统的整体数据处理能力,并增强了系统数据处理过程的可靠性和容错性。目前本文的研究成果已被应用在天文海量数据处理软件包的开发中并取得了良好效果,可以为其它用于海量数据处理的软件和系统的开发提供参考。
[Abstract]:MingantU Spectral Radioheliographer (MUSERA) can generate data of TB level every day. And we need to realize the real-time and offline processing of massive data, how to solve the problem of massive radio astronomical data processing in MUSER system. In order to meet the real-time and parallelism requirements of massive data processing, it is an important subject in the research and development of MUSER project. Mass data processing systems usually use message queue to improve the overall data processing capability of the system. This paper takes the massive radio astronomical data processing of MUSER system as the research background. In order to improve the overall data processing capability of MUSER system, the requirement of massive data processing in MUSER system is analyzed and the message queue technology is deeply understood. The main work of this paper is as follows: 1) after the selection of message queue technology, we choose to apply Kafka to MUSER system and decouple massive data processing into two parts: task publishing and task processing. To improve the parallelism and real-time of MUSER system, the multitasking parallel processing of MUSER system is realized. The overall data processing capability of MUSER system is improved. The system needs to realize load balancing to optimize task scheduling. But the native Kafka consumer load balancing algorithm can not meet the needs of MUSER system. This paper designs and implements an optimized consumer load balancing algorithm. Through the election of managers to control load balancing to optimize the process of consumer load balancing, further improve the overall data processing ability of MUSER system. In order to improve the fault tolerance of MUSER system, we use Kafka to deal with massive data. Consumption anomalies will reduce the accuracy of radio astronomical data processing results. This paper designs and implements the mechanism of consumer anomaly processing, through the dual-terminal log management. Exception handling and offset consistency check are used to improve the reliability and fault tolerance of data processing in MUSER system. In order to improve the accuracy of massive radio astronomical data processing results of MUSER system. The research results of this paper improve the overall data processing capability of MUSER system. The research results of this paper have been applied in the development of astronomical mass data processing software package and achieved good results. It can provide reference for the development of other software and systems for mass data processing.
【学位授予单位】：昆明理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：P161

【参考文献】