基于事件驱动的智能大数据实时处理平台的研究设计和基本实现

发布时间：2018-07-11 16:52

本文选题：Spark处理引擎 + 事件驱动架构　；参考：《北京邮电大学》2016年硕士论文

【摘要】：物联网的快速发展给人们的生活带来了极大的便利,同时也产生了大量蕴含着潜在信息价值的感知数据。感知数据具有数据量大、速度快、实时产生、动态变化等特点。如何更好地处理海量感知数据是当今物联网领域的研究热点之一。事件驱动架构(Event Driven Architecture)具有松耦合、高度集成、主动性和实时性等特点,能够较好地满足物联网应用的需求。目前物联网服务平台需要一个高效,高可用性的数据实时处理子系统。因此,本文的工作重点是结合事件驱动架构和Spark数据处理引擎构建一个松解耦、可拓展、高性能的数据实时处理平台,并设计实现图形化界面简化数据处理建模操作。具体工作如下:首先,本文设计和实现了一个数据接入模块,该模块从发布订阅系统获取事件并对其进行预处理;其次,针对数据处理建模过于复杂的问题,本文设计了图形化的数据处理建模模块,让用户能够直观简便地进行数据处理建模;再者,本文实现了平台界面模块,方便用户进行相关平台配置以及查看数据结果;另外,平台通过集成Spark引擎来提高计算速度;最后,本文对平台进行了相关测试,实验表明平台能够满足预期需求。本数据实时处理平台具有管理方便、数据处理建模简单、实时处理性能良好、基于事件驱动架构等特点,能够较好满足物联网服务平台的需求。
[Abstract]:The rapid development of the Internet of things brings great convenience to people's life, but also produces a large number of perceptual data with potential information value. Perceptual data has the characteristics of large amount of data, fast speed, real-time generation and dynamic change. How to better deal with mass perceptual data is one of the research hotspots in the field of Internet of things. Event driven Architecture (EventDriven Architecture) has the characteristics of loose coupling, high integration, initiative and real-time, which can meet the needs of Internet of things applications. At present, the Internet of things service platform needs a high-efficiency, high-availability data real-time processing subsystem. Therefore, the emphasis of this paper is to construct a loosely decoupled, extensible and high-performance real-time data processing platform based on event-driven architecture and Spark data processing engine, and to design a graphical interface to simplify the modeling operation of data processing. The specific work is as follows: firstly, this paper designs and implements a data access module, which acquires events from publish / subscribe system and preprocesses them. Secondly, the modeling of data processing is too complicated. This paper designs a graphical modeling module for data processing, which enables users to model data processing intuitively and easily. Furthermore, the platform interface module is implemented in this paper to facilitate users to configure relevant platforms and view data results; in addition, The platform improves the computing speed by integrating Spark engine. Finally, the platform is tested in this paper, and the experimental results show that the platform can meet the expected requirements. The real-time data processing platform has the advantages of convenient management, simple data processing modeling, good real-time processing performance and event-driven architecture, which can better meet the needs of the Internet of things service platform.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.44;TN929.5;TP311.13

【相似文献】