Storm平台下GPS车辆数据处理算法的实现

发布时间：2018-02-13 04:12

本文关键词： Storm GPS数据算法实时计算　出处：《西安电子科技大学》2015年硕士论文　论文类型：学位论文

【摘要】：伴随着经济的不断发展,物流行业如雨后春笋般蓬勃发展起来,物流车辆越来越多,物流车辆管理行业悄然兴起,我们利用便携式车载GPS实时采集车辆行驶过程中的时间、速度、经度、纬度等信息,根据客户的需求,设计算法,通过对这些数据进行分析、处理,来实现对物流车辆的远程监控、管理。面对GPS产生的海量数据,传统的处理方式已经力不从心,大数据处理技术的出现为其提供了一个很好的解决途径。Hadoop以及Storm从中脱颖而出,成为大数据处理的主要方法,Hadoop已经发展相当成熟,但其主要特点为对数据进行并行批处理,实时性很低,无法满足人们对于大数据处理实时性越来越高的要求,而Storm这个Apache基金会下的开源软件正好可以满足这种要求。Storm是一个分布式实时计算系统,它对于实时计算的意义类似于Hadoop对于批处理的意义。Storm的实时性体现在它以流的方式处理大数据,即Storm可以处理源源不断发送过来的消息,这些消息以数据元组为基本单位,形成一条有向无界的数据流,处理完之后将结果写入到某个存储中去。而其分布式特性体现在它的处理组件是分布式的,而且处理延迟极低,所以可以作为一个通用的分布式RPC框架来使用。所以,为了更好地管理车辆并且为其提供更优的服务,我们选择Storm这个平台来对GPS车辆大数据进行分析、处理。本文的核心内容是将两个GPS车辆数据处理算法在搭建好的Storm流式处理平台上进行实现。首先,对Storm的研究背景、意义、研究现状等进行了简单介绍,详细介绍了Storm的基本概念、特点、运行机制、系统架构、容错性能以及集群安装部署方法等内容,至此,可以对Storm有一个基础的了解。然后,详细介绍了两个GPS车辆数据处理算法——判断点是否在中国区域内的算法以及车辆轨迹匹配算法。最后,将这两个算法放在搭建好的Storm流式处理平台上进行实现,为其分别设计向集群提交的拓扑,从外部数据源读取数据进行处理,得到最终结果,以此来实现对车辆的监控。
[Abstract]:With the continuous development of economy, the logistics industry is booming, more and more logistics vehicles, logistics vehicle management industry quietly rising, we use portable vehicle GPS real-time acquisition of vehicle driving time, Speed, longitude, latitude and other information, according to the needs of customers, design algorithms, through the analysis of these data, processing, to achieve the remote monitoring of logistics vehicles, management. Facing the massive data generated by GPS, The traditional processing methods have been unable to meet their expectations, and the emergence of big data's processing technology has provided it with a very good solution. Hadoop and Storm have emerged from the fore, and become the main method of big data's handling, Hadoop has developed quite maturely. But its main characteristic is to carry on the parallel batch processing to the data, the real time is very low, cannot satisfy the people to big data processing real-time request which is more and more high. And Storm, the open source software of the Apache Foundation, just meets this requirement. Storm is a distributed real-time computing system. Its meaning for real-time computing is similar to that of Hadoop for batch processing. A directed unbounded data stream is formed, and the results are written to a storage after processing. Its distributed nature is that its processing components are distributed and the processing latency is extremely low. So it can be used as a general distributed RPC framework. Therefore, in order to better manage vehicles and provide better service for them, we choose Storm as a platform to analyze GPS vehicle big data. Processing. The core of this paper is to implement two GPS vehicle data processing algorithms on the Storm flow processing platform. Firstly, the research background, significance and research status of Storm are briefly introduced. The basic concept, characteristics, running mechanism, system architecture, fault-tolerant performance and cluster installation and deployment methods of Storm are introduced in detail. At this point, we can have a basic understanding of Storm. This paper introduces in detail two GPS vehicle data processing algorithms, which are the algorithms for judging whether the points are in the Chinese region and the vehicle trajectory matching algorithm. Finally, the two algorithms are implemented on a Storm flow processing platform. The topology to be submitted to the cluster is designed respectively, and the data is read from the external data source for processing, and the final result is obtained, so as to realize the monitoring of the vehicle.
【学位授予单位】：西安电子科技大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TP311.13

【相似文献】