基于Hadoop的离线视频数据处理技术研究与应用

发布时间：2018-06-23 14:55

本文选题：大数据处理 + 视频离线处理　；参考：《北京邮电大学》2014年硕士论文

【摘要】：当前,智慧城市成为信息时代城市建设的一个基本目标,智能视频安防监控是其中重要一环。视频监控系统已广泛使用于各行各业,监控视频数据已成为一类典型的大数据,传统的视频收集与回放已不能满足人们对视频监控的需求,我们希望从视频图像提取出有效的信息,提供有效的治安防控业务信息,因此,如何对监控视频大数据进行高效的处理成为一个重要研究课题。本文首先深入分析了Hadoop框架中的三个重要组成部分,即分布式存储系统HDFS、分布式计算框架MapReduce和分布式数据库HBase,并总结了目前一些常用的基于内容的视频处理方法,说明了目前离线视频处理方法的瓶颈和不足。在分析视频处理特点的基础上,提出并实现了一种基于Hadoop MapReduce计算框架的分布式离线视频处理方法,通过设计Hadoop视频处理相关方法、接口,使Hadoop MapReduce可以像处理文本文件和二进制文件那样处理视频数据,解决了Hadoop MapReduce不能直接处理视频数据的问题,这样,开发人员在基于Hadoop对视频数据进行并行处理时,就可以将更多精力集中在视频处理的核心算法上。同时,针对视频处理时间与视频复杂度相关这一特点,本文对Hadoop HDFS的数据分布进行了重分布设计与实现,使Hadoop MapReduce在进行此类监控视频大数据处理时,系统整体性能有进一步优化。在此基础上,论文对Hadoop MapReduce的数据类型进行了进一步的分析和设计,并基于此实现了离线视频数据处理系统,集成了分布式视频转码、分布式视频摘要和分布式人员检索三个应用。测试结果表明,利用本系统完成海量视频数据处理所需时间开销大大减少,通过对HDFS的数据分布进行重分布优化,减少了系统I/O,进一步提高了Hadoop MapReduce处理视频应用的效率。
[Abstract]:At present, intelligent city has become a basic goal of urban construction in the information age. Intelligent video security monitoring is an important part of it. Video surveillance system has been widely used in all walks of life. Monitoring video data has become a kind of typical large data. Traditional video collection and replay can not meet people's demand for video surveillance. We want to extract effective information from video images and provide effective information on public security and prevention and control. Therefore, how to efficiently handle large data of monitoring video has become an important research topic.
This paper first analyzes three important components in the Hadoop framework, namely, distributed storage system HDFS, distributed computing framework MapReduce and distributed database HBase, and summarizes some commonly used video processing methods based on content. It illustrates the bottleneck and shortcomings of the current off-line frequency processing method. On the basis of rational characteristics, a distributed off-line video processing method based on Hadoop MapReduce computing framework is proposed and implemented. By designing Hadoop video processing related methods and interface, Hadoop MapReduce can handle visual frequency data like text files and binary files, and the Hadoop MapReduce can not be dealt with directly. The problem of video data is so that the developer can focus more on the core algorithm of video processing when it is based on the parallel processing of video data based on Hadoop.
At the same time, in view of the feature of video processing time and video complexity, the redistribution of the data distribution of Hadoop HDFS is designed and implemented, so that the overall performance of the system is further optimized when Hadoop MapReduce is processed for large data processing of this kind of monitoring video.
On this basis, the paper makes a further analysis and design of the data types of Hadoop MapReduce. Based on this, the off-line video data processing system is implemented, which integrates three applications of distributed video transcoding, distributed video summarization and distributed personnel retrieval.
The test results show that the time cost of using this system to complete mass video data processing is greatly reduced. By redistributing the data distribution of HDFS, the system I/O is reduced, and the efficiency of Hadoop MapReduce processing video application is further improved.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TN948.6;TP311.13

【参考文献】