基于深度学习的人类行为识别和视频描述生成

发布时间：2018-06-14 23:30

本文选题：行为识别 + 视频描述生成　；参考：《电子科技大学》2017年硕士论文

【摘要】：视频智能分析一直是计算机视觉领域的研究热点。其中涉及到不同的研究问题,包括视频语义分割,视频追踪,视频检索,行为识别以及视频描述生成等等。为了更进一步构建视频内容与高层语义之间的桥梁,本文围绕两个具体的视频应用进行深入研究:视频中的行为识别和视频描述生成。具体而言,我们把行为识别视作低层语义分类问题。而将视频描述问题视作为高层语义生成问题。因为该任务要求同时理解视觉内容和自然语言。而针对这些问题的挑战,本文具体研究两个问题:1)如何构建一个算法计算视频对应的模式。2)如何构建一个有效的计算框架来架起视频内容与自然语言的桥梁。对于视频中的行为识别而言,传统的方法将此问题化为多分类问题,并且提出了不同的视频特征提取方法。然而,传统的方法基于低层信息进行提取,比如从视觉纹理信息或者视频中的运动估计值。由于提取的信息单一,不能很好的代表视频内容,进而导致所优化的分类器并不是最优的。而作为深度学习中的一项技术,卷积神经网络将特征学习和分类器学习融合为一个整体,并且成功应用在视频中的行为识别方面。然而,当前提出的并应用在行为识别方面的卷积神经网络框架仍具有三种限制:1)输入网络的视频的空间尺寸必须固定大小;2)输入网络的视频的时长固定;3)网络提取短时序结构的特征。这使得网络模型应用在极强的限制条件下,不利于现实场景下的应用扩展。为了解决以上的问题,本文提出一种基于3D卷积网络的端到端识别模型。该模型实现在任意尺度和时长的视频条件下进行行为识别。具体而言,首先将一个视频划分为一系列连续的视频片段。然后,将连续的视频片段输入到由卷积计算层和时空金字塔池化层组成的3D神经网络得到连续的视频片段特征。然后通过长短记忆模型计算全局的视频特征作为行为模式。我们在UCF101,HMDB51和ACT三个通用的数据集上评估提出的模型。实验结果显示,和目前流行的2D或3D为基础的神经网络模型相比,提出的方法在识别性能上得到了提升。在视频描述生成方面,以编码-解码为基础的框架已经得到了广泛的应用。最近,时序注意力机制已经被提出并且被证明能够提升以编码-解码为基础的描述生成模型的性能。然而,时序注意力机制只解决了视频内容的选取。对于语句的上下文则是由先验语义而定。然而,在视频描述生成这个方面,目前的方法没有同时考虑时序注意力机制和先验语义建模。为了解决这个问题,本文提出一个新的端到端的神经网络模型,能够将高层的视觉语义概念融入到时序注意力机制,并促进生成更准确的视频描述。在提出的框架中,编码神经网络模型用于提取视频的视觉特征,并且通过该特征预测语义概念。同时,解码神经网络根据视觉特征与语义信息来生成连贯的自然语言句子。具体而言,解码神经网络结合了视觉特征和语义表达特征。并且将语义信息和注意力机制嵌入到GRU神经网络单元中去更加准确的学习句子的生成。本文在两个代表性的数据集上(MSVD和MSRVTT)验证提出的框架。实验结果显示提出的网络模型在BLEU和METEOR两个评价标准上,比以往的方法得到更好的性能评估。
[Abstract]:Video intelligence analysis has been a hot topic in the field of computer vision. It involves different research issues, including video semantic segmentation, video tracking, video retrieval, behavior recognition and video description generation. In order to further build a bridge between video content and high level semantics, this paper focuses on two specific videos. Conduct in-depth study: behavior recognition and video description generation in video. Specifically, we regard behavior recognition as a low level semantic classification problem. Video description is considered as a high-level semantic generation problem. The task requires simultaneous understanding of visual content and natural language. Two problems: 1) how to build an algorithm for computing video corresponding to.2) how to build an effective framework to build a bridge between video content and natural language. For the behavior recognition in video, the traditional method turns this problem into a multi classification problem, and presents different methods of video feature extraction. The traditional method is based on low level information, such as the motion estimation from visual texture information or video. Because the extracted information is single, it can not represent the video content well, and the optimized classifier is not optimal. As a technique in depth learning, the convolution neural network will learn the feature and the feature. Classifier learning is integrated into a whole and is successfully applied to behavior recognition in video. However, the current convolution neural network framework which is proposed and applied to behavior recognition still has three limitations: 1) the space size of the video in the input network must be fixed in size; 2) the time length of the video in the input network; 3) network extraction. In order to solve the above problems, this paper proposes an end to end recognition model based on 3D convolution network. This model implements the behavior recognition under arbitrary and long time video conditions. First, a video is divided into a series of continuous video clips. Then, the continuous video clips are input into the 3D neural network composed of the convolution computing layer and the spatio-temporal Pyramid pool layer to get the continuous video clip features. Then the video features of the global video are calculated by the long and short memory model as the behavior pattern. We are in the UCF101, The experimental results show that the proposed method has been improved in recognition performance compared with the current popular 2D or 3D based neural network models. In the aspect of video description generation, the framework based on encoding and decoding has been widely used. Recently, time series has been applied to HMDB51 and 3D based neural network models. The attention mechanism has been proposed and proved to be able to improve the performance of the description generation model based on the coding decode. However, the time series attention mechanism only solves the selection of video content. The context of the statement is determined by a priori semantics. However, the current method is not tested at the same time in the aspect of video description generation. In order to solve this problem, this paper proposes a new end to end neural network model, which can integrate the high-level visual semantic concepts into the time series attention mechanism and promote more accurate video description. In the frame, the coded neural network model is used to extract video. The visual features are predicted and the semantic concepts are predicted by this feature. At the same time, the neural network is decoded to generate coherent natural language sentences based on visual features and semantic information. In particular, the decoded neural network combines the visual features and semantic expression features. And the semantic information and attention mechanism are embedded into the GRU neural network unit. To more accurately learn the generation of sentences. This paper validates the proposed framework on two representative data sets (MSVD and MSRVTT). The experimental results show that the proposed network model is better performance evaluation than previous methods on the two evaluation criteria of BLEU and METEOR.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.41;TP181

【相似文献】