交通安防大数据的实时快速检索关键技术研究

发布时间：2018-04-24 08:18

本文选题：智慧交通 + 检索　；参考：《浙江大学》2017年硕士论文

【摘要】：随着城市化进程加快,城市人口迅速增加,带来了日益严重的交通安全问题。“智能交通”可以通过城市监控所采集的大量数据,对交通情况、城市安全进行分析,协助城市管理者管理城市的交通与安全。HBase数据库可以大规模存储车辆信息,但是无法快速实时进行检索。本研究主要面向对HBase中存储的过车记录进行快速实时检索,提出通过降低网络开销、硬盘读取开销的优化方法。首先,在HBase中建立二级索引降低检索中扫描的数据量,计算二级索引来获得符合检索条件的记录。实验结果表明,返回第一条过车记录的检索响应时间相比无二级索引时降低45倍。其次,二级索引从存储端传输至客户端的网络传输开销时间占总检索时间的20%。为了降低网络传输开销,基于近数据计算理论,通过HBase协处理器构建加速框架将二级索引的获取和计算移动到存储节点进行。在集成了该加速框架后,网络开销降低了 12倍,检索响应时间加速比为1.4。最后,在检索时发现存储节点从硬盘读取二级索引的时间占总时间开销的70%。为了降低硬盘读取时间开销,根据二级索引离散度的不同,组合多种无损压缩算法对二级索引进行压缩来降低硬盘读取时间,并在HBase协处理器中实现压缩后索引的计算。实验结果表明,硬盘读取时间降低73%,检索响应时间降低 80%。
[Abstract]:With the acceleration of urbanization and the rapid increase of urban population, traffic safety problems are becoming more and more serious. "Intelligent Transportation" can analyze the traffic situation and urban safety through a large amount of data collected by urban monitoring, and help city managers manage the city's traffic and safety. HBase database can store vehicle information on a large scale. But there is no quick and real-time retrieval. This research mainly aims at the fast real-time retrieval of the passing records stored in HBase, and proposes an optimization method to reduce the network overhead and the hard disk reading overhead. Firstly, a secondary index is established in HBase to reduce the amount of data scanned in the retrieval process, and the secondary index is calculated to obtain the records that meet the retrieval criteria. The experimental results show that the retrieval response time of returning the first passing record is 45 times lower than that without the second level index. Secondly, the network transmission overhead from storage to client accounts for 20% of the total retrieval time. In order to reduce the network transmission overhead, based on the theory of near data computing, an accelerated framework is constructed by HBase coprocessor to move the acquisition and computation of the secondary index to the storage node. With the integration of the acceleration framework, the network overhead is reduced by 12 times, and the retrieval response time speedup is 1.4. Finally, it is found that the storage node reads the secondary index from the hard disk in 70% of the total time cost. In order to reduce the read time cost of the hard disk, according to the difference of the dispersion of the secondary index, several lossless compression algorithms are combined to compress the secondary index to reduce the read time of the hard disk, and the computation of the compressed index is realized in the HBase coprocessor. The experimental results show that the hard disk reading time is reduced by 73 and the retrieval response time is reduced by 80 percent.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：U495;U492.8

【参考文献】