Gearpump流处理引擎的Storm兼容性模块的设计与实现

发布时间：2018-03-11 08:10

本文选题：大数据流处理　切入点：流处理引擎　出处：《南京大学》2016年硕士论文　论文类型：学位论文

【摘要】：随着大数据流处理技术的不断成熟和广泛运用,越来越多的大数据流处理平台涌现了出来。大数据流处理能够对动态产生的数据进行实时计算并及时反馈结果,在数据的有效时间内挖掘其价值,因此成为了应用于诸多场景的重要技术。Storm是目前在业界被广泛使用的一个流处理平台,但它也有一定的局限性。Intel最新开源的流处理引擎Gearpump解决了Storm的一些局限性,为用户提供一个更加优秀的流处理平台。为了使用户能够以较低的成本体验到Gearpump的优良特性,我们设计并开发了基于Gearpump流处理引擎的Storm兼容性模块,使Storm的应用能够直接运行在Gearpump平台上。为了实现上述的Gearpump流处理引擎的Storm兼容性模块,本文首先介绍了流处理平台的相关背景及发展现状,并对目前主流的流处理平台Storm、 Samza、Spark等进行了调研。紧接着介绍了本项目的关键技术,并对两个与本项目密切相关的流处理平台Storm和Gearpump进行了深入的研究和分析,提出了基于Gearpump流处理平台搭建一个Storm兼容层的需求。基于之前的研究工作,分析了项目需求,重点阐述了提交应用和拓扑图转换等功能需求和用例描述,并对系统的三个子模块GearpumpNimbus子模块、GearpumpStormClient子模块和拓扑图转换子模块进行了详细的设计。最后,给出了各个子模块的具体实现细节。本文所实现的Gearpump的Storm兼容性模块成功的将Storm应用运行在Gearpump上,并且保证了应用运行时的高性能,让Storm用户在Gearpump平台上体验到了其特点。
[Abstract]:With the continuous maturity and wide application of large data flow processing technology, more and more large data flow processing platforms have emerged. Large data stream processing can calculate the data generated dynamically in real time and feedback the results in time. Mining its value within the effective time of data has become an important technology used in many scenarios. Storm is currently widely used as a stream processing platform in the industry. But it also has some limitations. Intel's latest open source stream processing engine, Gearpump, solves some of the limitations of Storm and provides a better streaming platform for users. In order to enable users to experience the superior features of Gearpump at a lower cost, We have designed and developed a Storm compatibility module based on Gearpump stream processing engine, which enables the application of Storm to run directly on the Gearpump platform. In order to realize the Storm compatibility module of the Gearpump stream processing engine mentioned above, This paper first introduces the background and development status of the flow processing platform, and investigates the current mainstream stream processing platforms Storm, Samzaan Spark, etc. Then it introduces the key technologies of this project. In addition, two flow processing platforms, Storm and Gearpump, which are closely related to this project, are studied and analyzed, and the requirement of constructing a Storm compatible layer based on Gearpump flow processing platform is put forward. Based on the previous research work, the requirements of the project are analyzed. The functional requirements and use case description of submission application and topology diagram transformation are described in detail, and the design of three sub-modules of the system, GearpumpNimbus submodule, GearpumpStormStormClient module and topology transformation submodule, are given in detail. The implementation details of each sub-module are given. The Storm compatibility module of Gearpump implemented in this paper successfully runs the Storm application on Gearpump, and ensures the high performance of the application runtime, which enables Storm users to experience its characteristics on the Gearpump platform.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.52

【相似文献】