基于Kafka的大规模流数据分布式缓存与分析平台

发布时间：2018-08-09 10:45

【摘要】：近年来,随着信息技术与互联网应用的不断发展,全球数据总量也在呈现爆炸式的增长,大数据时代即将来临。这将不仅为科学研究领域带来巨大变革,也将深远地影响到未来我们生活的方方面面。现阶段,在大数据分析与计算领域,由于分布式集群架构低成本、高计算能力、良好的可扩展性等特点获得了越来越广泛的应用。于此同时,分布式集群架构中所计算和分析的数据结构也越来越多样化。近年电子商务、物联网,互联网金融等领域的应用不断发展,在多数分布式集群中同时存在着监控端传输的动态流数据和系统生成的运行时日志文件。这种情况下,由于不同特点的数据所适合分析算法和计算方式也有所不同,例如流数据处理过程关于实时性和拓扑结构多样性的要求,大规模数据批量处理过程中的系统吞吐量和资源利用率的要求。而现有的主流分布式集群系统通常只适合对一种特定的数据进行分析,例如Hadoop[19][21]、Storm[22]以及S4[23]等,而无法适应多种类型数据结构并存情况。本文创新性的提出了基于kafka的大规模流数据分布式缓存与分析平台。该平台的设计目标是组织和缓存系统输入的大规模流数据。并分别设计在线流数据处理和离线批处理多种方式的处理单元,依据不同数据类型选择合适的方式进行分析运算。总结该缓存与分析平台的特点,主要分为以下几个方面:(1)采用分布式消息系统作为大规模流数据的缓存,提高了平台对动态流数据输入数据量突发性变化的适应能力。(2)设计并实现在线实时处理单元及离线批处理单元,分别处理集群中不同特点的数据,以满足不同类型的数据对计算实时性和系统吞吐量不同方面的需求。(3)整个平台采用集中式的管理方式,不同模块、不同处理单元中的节点信息统一同步到管理模块,以实现平台节点信息的全局一致性。本文详细介绍了平台的总体架构设计,将系统分为三个部分,分别实现缓存订阅、在线实时处理以及系统管理等功能。并基于此设计实现了基于kafka的大规模流数据分布式缓存与分析平台模型。最后验证了平台的可用性,可扩展性及高效性等特点。本文希望能通过该平台的设计与实现过程,给分布式计算集群的搭建和大规模流数据处理过程提供新的思路和方法。也希望能够通过进一步的努力,不断完善平台模型,将该平台用于实际生活、生产、以及研究过程中。
[Abstract]:In recent years, with the continuous development of information technology and Internet applications, the global data volume is also explosive growth, the big data era is coming. This will not only bring about great changes in the field of scientific research, but will also have a profound impact on all aspects of our lives in the future. At present, in the field of big data analysis and computing, the distributed cluster architecture has been applied more and more widely because of its low cost, high computing power and good scalability. At the same time, the data structures calculated and analyzed in the distributed cluster architecture are more and more diversified. In recent years, the applications of electronic commerce, Internet of things, Internet of Finance and so on have been continuously developed. In most distributed clusters, there are dynamic stream data transmitted by monitoring terminal and runtime log files generated by the system at the same time. In this case, due to different characteristics of the data suitable for analysis algorithms and calculation methods are different, such as the flow of data processing process for real-time and topology diversity requirements, Requirements of system throughput and resource utilization during mass processing of large-scale data. However, the existing mainstream distributed cluster systems are generally suitable for the analysis of a specific data, such as Hadoop [19] [21] Storm [22] and S4 [23], but can not adapt to the coexistence of many types of data structures. In this paper, a kafka-based distributed cache and analysis platform for large-scale stream data is proposed. The platform is designed to organize and cache large-scale stream data input from the system. The processing units of on-line stream data processing and off-line batch processing are designed, and the analysis and operation are carried out according to different data types. The characteristics of the cache and analysis platform are summarized, which are divided into the following aspects: (1) the distributed message system is used as the cache of large-scale stream data. It improves the adaptability of the platform to the sudden change of the data input data from the dynamic flow. (2) the on-line real-time processing unit and the off-line batch processing unit are designed and implemented to process the data with different characteristics in the cluster, respectively. In order to meet the requirements of different types of data for real-time computing and system throughput. (3) the whole platform adopts centralized management mode, different modules, different processing unit node information synchronization to the management module, In order to realize the global consistency of the platform node information. This paper introduces the overall architecture of the platform in detail. The system is divided into three parts: cache subscription, online real-time processing and system management. Based on this design, the distributed cache and analysis platform model of large scale stream data based on kafka is implemented. Finally, the usability, extensibility and efficiency of the platform are verified. Through the design and implementation of the platform, this paper hopes to provide new ideas and methods for the construction of distributed computing clusters and large-scale data processing. It is also hoped that through further efforts, the platform model can be improved and used in real life, production, and research process.
【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】