支持多源异构流数据汇集的通信服务器设计与实现

发布时间：2018-05-14 13:14

本文选题：流数据 + 数据通信　；参考：《北方工业大学》2016年硕士论文

【摘要】：随着物联网技术的普及,催生了很多新的应用,同时给传统行业带来了变化,例如智能家居、智能交通、智慧城市等。不同行业的行为活动已经被转变为动态流数据,这些数据具有实时性、并发性、异构性等新特点,传统的处理方法和处理模型已经不能够满足于动态流数据的接入需求。流数据接入的主要难点在于数据种类多样和数据来源广泛：例如有GPS数据、交通车辆数据、传感器采集的数据以及其他感知数据等类型；数据来源包括交通、农业、医疗、电商等各个领域来自于大量的前端传感器和不同行为转化的数据。在上述数据接入的技术需求下,产生了两方面的问题：动态流数据怎么被接入到云中心,不同类型的动态流数据如何传输和大量前端的异构数据怎么接入；如何满足流数据通信服务器对于数据的高效解析和分发,为了应对大量数据的接入和支撑其它业务的处理,需要高效的解析数据和分发数据。本文针对现在缺少有效的流数据汇集系统,在掌握了网络通信协议、异构流数据汇集技术以及异步10技术后,设计并实现了一个支持多源异构流数据汇集的通信服务器,我们称之为流数据通信服务器。在云环境下可以支持海量数据的接入、解析和分发。并且为用户提供web端,可以监测数据变化,同时对数据进行持久化,以备其它应用可以使用流数据。论文的主要内容包括：(I)设计了一种支持异构流数据汇集的数据通信协议,包含了发送的数据帧格式、响应的数据帧格式以及增强可靠性的一些设计。把不同来源、不同类型的数据按照数据通信协议的格式打包再接入系统,不仅实现了数据通信而且能够支持异构流数据的接入。(2)采用异步10方式实现了支持高并发的流数据通信服务器,本文主要使用了Libevent和多线程连接池等技术来实现高并发机制。在接入大量数据后能够高并发的对数据进行解析和分发来保证系统的性能,同时能够将处理后消息包转发给消息服务器实现消息中转。(3)开发了一套物联网感知数据托管服务应用原型系统,在本文已实现的流数据通信服务器基础上设计的。能够实现用户对设备和数据的管理,为云环境下的不同用户的不同类型的感知数据汇集提供了新的服务支撑模式。此外,经实验测试,系统在普通服务器配置下可达到3000并发长连接(即3000个终端设备下每秒3000条数据的接收能力)
[Abstract]:With the popularization of the Internet of things technology, many new applications have been spawned, at the same time brought changes to the traditional industries, such as smart home, intelligent transportation, intelligent city and so on. The behavioral activities of different industries have been transformed into dynamic flow data. These data have some new characteristics such as real-time, concurrency, heterogeneity and so on. The traditional processing methods and processing models can no longer meet the access requirements of dynamic flow data. The main difficulties in accessing stream data are the variety of data and the wide range of data sources, such as GPS data, traffic vehicle data, sensor data and other perceptual data, etc. The data sources include transportation, agriculture, medicine, etc. E-quotient and other fields come from a large number of front-end sensors and different behavior conversion data. Under the technical requirements of the above data access, there are two problems: how to connect the dynamic stream data to the cloud center, how to transmit the different types of dynamic flow data and how to access a large number of heterogeneous data in the front end; How to satisfy the efficient data parsing and distribution of streaming data communication server, in order to deal with the access of a large number of data and support the processing of other services, it is necessary to efficiently parse and distribute data. Aiming at the lack of effective streaming data collection system, after mastering the network communication protocol, heterogeneous stream data collection technology and asynchronous 10 technology, this paper designs and implements a communication server that supports multi-source heterogeneous stream data collection. We call it streaming data communication server. In the cloud environment can support massive data access, parsing and distribution. It can monitor the data change and persist the data in case other applications can use stream data. The main contents of this paper are as follows: (1) A data communication protocol supporting heterogeneous stream data collection is designed, which includes the data frame format sent, the response data frame format and some designs to enhance reliability. Packaging different sources and types of data into the system in the format of the data communication protocol, This paper not only realizes data communication but also supports the access of heterogeneous stream data. It uses asynchronous 10 mode to realize the high concurrency stream data communication server. This paper mainly uses Libevent and multi-thread connection pool to realize the high concurrency mechanism. After accessing a large amount of data, the data can be parsed and distributed in high concurrency to ensure the performance of the system. At the same time, we can forward the processed message packet to the message server to realize the message transfer. We have developed a prototype system of Internet of things aware data hosting service, which is designed on the basis of the stream data communication server which has been implemented in this paper. It can realize the user's management of the device and data, and provide a new service supporting mode for the different types of perceptual data collection of different users in the cloud environment. In addition, experimental tests show that the system can achieve 3000 concurrent long connections (i.e., the receiving capacity of 3000 data per second under 3,000 terminal devices) under the common server configuration.
【学位授予单位】：北方工业大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】