基于Hadoop的海量城市交通流数据分布式存储与分析研究

发布时间：2018-04-07 17:38

本文选题：智能交通　切入点：Hadoop　出处：《扬州大学》2015年硕士论文

【摘要】：随着智能交通基础建设的快速发展,城镇居民收入水平逐步提高,城市汽车拥有量大幅度增加。遍布每个城市道路的感应线圈、卡口断面系统,能够及时地采集、记录、汇总并上传监控数据。但是由于城市道路交通流存在着数据量大、实时性高等特点,传统的数据存储与处理技术存在着数据结构与数据存储容量无法灵活扩展、分布式并行数据挖掘难、高容错恢复能力差等问题。如何将海量的交通流数据实时地上传、汇总和存储利用,以及如何对数据进行统计挖掘成为一个较大的难题。以Hadoop为代表的大数据技术成为解决这一系列问题的有效手段之一。基于现阶段城市交通发展带来的数据存储与分析等突出问题,本文通过对基于Hadoop的MapReduce、HBase等大数据技术的研究,提出了相应的解决方案,其主要研究工作和成果如下：(1)本文提出了基于Hadoop的交通流数据存储与分析总体架构。将架构分为5个层面：数据采集层、硬件平台层、数据存储与计算层、挖掘分析层和应用服务层,同时研究与设计了节点在故障或宕机情况下,Hadoop集群具有高容错恢复能力的可用性方案。(2)本文提出了基于HBase的海量交通流数据分布式存储方案。根据交通流数据特点与处理应用需求,设计了可解决“热点”问题的交通流数据表行健结构。同时研究了HBase的协处理器,设计了用于针对列查询的快速数据检索的二级索引表。(3)本文还根据交通车流量与密度的关系,设计了流量与密度计算模型,提出了基于MapReduce的流量密度计算的并行化实现,解决了海量交通流数据情况下的流量、密度快速计算难题。同时,采用K近邻非参数回归算法来预测短时交通流,通过对K近邻状态向量、距离度量方式、近邻个数以及预测算法的选择及研究,提出了基于MapReduce的KNN预测短时交通流的并行化实现,加快K最近邻算法的搜索速度,实现对短时交通流的定时预测。(4)最后,根据总体架构应用层需求,基于Hadoop平台,构建并实现了城市道路交通流数据分析系统。本文对系统进行了详细的功能模块设计,并实现了对交通流量进行实时监测、海量数据分析的图形化展示等功能。
[Abstract]:With the rapid development of intelligent transportation infrastructure, the income level of urban residents has gradually increased, and the number of urban car ownership has increased significantly.Induction coil and bayonet section system all over every city road can collect, record, collect and upload monitoring data in time.However, due to the characteristics of large amount of data and high real-time performance in urban road traffic flow, the traditional data storage and processing technology can not extend data structure and storage capacity flexibly, and distributed parallel data mining is difficult.Poor recovery ability of high fault tolerance and so on.How to upload, aggregate, store and utilize massive traffic flow data in real time, and how to mine the data statistically has become a big problem.Big data technology, represented by Hadoop, has become one of the effective means to solve this series of problems.Based on the outstanding problems of data storage and analysis brought about by the development of urban traffic at present, this paper puts forward the corresponding solutions through the research of big data technology such as MapReduceHBase based on Hadoop.The main research work and results are as follows: (1) this paper proposes an overall framework of traffic flow data storage and analysis based on Hadoop.The architecture is divided into five layers: data acquisition layer, hardware platform layer, data storage and computing layer, mining analysis layer and application service layer.At the same time, we study and design the availability scheme of Hadoop cluster with high fault-tolerant recovery ability in the event of failure or downtime) this paper proposes a distributed storage scheme for massive traffic flow data based on HBase.According to the characteristics of traffic flow data and the requirement of application, the traffic flow data table is designed to solve the "hot spot" problem.At the same time, the coprocessor of HBase is studied, and a two-level index table for fast data retrieval for column query is designed. In this paper, according to the relationship between traffic flow and density, the calculation model of traffic flow and density is also designed.A parallel implementation of traffic density calculation based on MapReduce is proposed, which solves the problem of fast calculation of traffic density in the case of massive traffic flow data.At the same time, the K-nearest neighbor nonparametric regression algorithm is used to predict the short-term traffic flow. The selection and research of K-nearest neighbor state vector, distance measurement, number of nearest neighbors and prediction algorithm are carried out.In this paper, the parallel implementation of short time traffic flow prediction with KNN based on MapReduce is proposed, which speeds up the search speed of K nearest neighbor algorithm, and realizes the timing prediction of short time traffic flow. Finally, according to the requirements of the application layer of the overall architecture, it is based on Hadoop platform.The data analysis system of urban road traffic flow is constructed and implemented.In this paper, the function module of the system is designed in detail, and the functions of real-time monitoring of traffic flow and graphical display of mass data analysis are realized.
【学位授予单位】：扬州大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：U495;TP311.13

【参考文献】