基于序列深度学习的视频分析：建模表达与应用

发布时间：2018-09-01 11:47

【摘要】：近年来,视频数据呈现爆炸式增长。如此大量的视频数据在存储、识别、共享、编辑、生成等过程中都需要精准的视频语义分析技术。深度学习自从2012年以来极大地推动了计算机视觉领域向前发展,使得大规模视频数据的分析成为可能。因此,利用深度学习进行视频语义分析的研究也就成为了必然的选择。目前基于深度学习的视频语义分析基本上分为两大步:1)利用卷积神经网络提取出各帧的视觉特征表达;2)利用长短时递归神经网络(LSTM)对该特征序列进行学习,并表达为相应的语义类别或语义描述。本文在对现有视频语义分析技术进行全面调研和总结的基础之上,对视频语义分类和视频语义描述两种任务深度学习模型中存在的经典问题进行了充分的研究。从视频帧的视觉特征表达的角度,分别提出了连续Dropout算法、参数对图像变换稳健的卷积神经网络、结构对图像变换稳健的卷积神经网络.从特征融合、特征提取的关键角度进一步改善现有卷积神经网络的性能。与此同时,针对序列学习中多层递归神经网络训练难、训练慢的问题,提出了以无监督逐层贪婪式学习的方式进行预训练,有效提升模型性能和模型的训练效率。更进一步地,针对现有视频序列到单词序列的单向映射框架的局限性,创造性地提出了基于潜在语义表征的多向序列学习算法,为基于视频和句子序列的分析提供了新的视角。本文的主要工作和创新点总结如下:连续Dropout Dropout已经被证明是一种有效的训练深度卷积神经网络的算法,其主要的思想为通过在一个大规模的卷积神经网络中屏蔽一些原子,可以同时训练多个弱分类器,在预测阶段.可以将这些弱分类器的结果综合起来,有效提升卷积神经网络的性能。人脑中的神经元的突触的信号传递强度是连续随机的而不是离散随机的,受这种现象的启发,我们将传统的二进制Dropout扩展到连续Dropout。一方面,连续Dropout比传统的二进制Dropout更接近人脑中神经元的激活特性。另一方面,我们证明连续Dropout具有避免特征检测器共同适应的特性。这表明我们可以在预测阶段融合更多的独立特征检测器提取出来的特征,从而得到更好的结果。参数稳健的卷积神经网络卷积神经网络(CNN)已经在许多视觉任务上获得了最好的结果,目前几乎所有的视觉信息都要用到卷积神经网络进行处理。然而,目前的CNN模型仍然表现出对图像空间变换的稳健性比较差。直观地,具有有足够的层数和参数卷积神经网络通过卷积(矩阵乘法和非线性激活)和池操作的分层组合应该能够学习从变换输人图像到变换不变表示的鲁棒映射。我们提出在训练阶段随机变换(旋转、缩放和平移)CNN的特征图。这可以降低CNN模型对训练图像的特定旋转,缩放和平移的依赖。相反,每个卷积核将学习到对于其输人特征图的多种变换组合中不变的特征。这样,我们不需要对优化过程和训练图像添加任何额外的监督信息或对输入图像进行修改。通过实验进一步验证了通过随机变换进行学习的CNN对于输入图像的变换更加不敏感,在小规模图像识别,大规模图像识别和图像检索上都提升了现有卷积神经网络的性能。结构稳健的卷积神经网络卷积神经网络(CNN)已经在许多视觉识别任务中展现了最好的性能。然而,卷积和池化操作的组合对输人中有意义目标的局部位置变化只显示出较小的不变性。有时.一些网络使用数据增强(Data augmentation)的方法来训练网络,以将这种不变性编码到网络参数中,但这限制了模型学习目标内容的能力。一种更有效的方式是将对平移或旋转的不变性添加到网络架构中,从而减轻了模型对参数学习的需要。为了使模型能够专注于学习其所描述的对象,而不受其位置等影响,我们提出在特征响应图中,通过对局部区块进行排序,然后再输入进下一层中。当区块重排序结合卷积和池操作时,我们对于处于不同位置的输入图像中的目标,获得了一致的表达。我们证明了所提出的区块排序模块可以提高CNN对许多基准任务的性能,包括MNIST数字识别、大规模图像识别和图像检索等。序列的深层递归神经网络学习递归神经网络(RNNs)特别是视频分析中常用的长短时记忆网络(LSTM)的最新发展已经显示出其对序列数据进行建模的潜力,特别是在计算机视觉和自然语言处理领域。然而,对拥有多层LSTM的深层网络的训练效果相对于浅层网络并没有得到期望的提升,同时收敛速度也比较慢。这种困难源于LSTM的初始化方法,其中基于梯度的优化通常会收敛于较差的局部解。针对此问题,我们探讨了无监督的LSTM初始化预训练机制,充分利用无监督预训练的作用,指导后续的监督训练。在本文中,我们提出了一种新颖的基于编码器-解码器的学习框架,以贪婪式的逐层训练的方式初始化多层LSTM,其中每个新增的LSTM层被训练以保留上一层得到的表示中的主要信息。使用我们的预训练方法训练的多层LSTM胜过随机初始化训练的的LSTM,在回归(加法问题),手写数字识别(MNIST),视频分类(UCF-101)和机器翻译WMT'14)等任务上都取得了更好的效果。此外,使用贪婪的逐层训练方法还将多层LSTM的收敛速度提高了 4倍。基于共享隐表征的序列到序列学习序列学习是深度学习的流行研究领域,如视频字幕和语音识别。现有方法通过首先将输入序列编码为固定大小的向量.然后从向量解码目标序列来将该学习建模为映射过程。虽然简单直观,但是这种映射模型是任务相关(task-specific)的,不能直接用于不同的任务。在本文中,我们为通用和灵活的序列到序列学习提出了一个星状框架,其中不同类型的媒体内容(外围节点)可以被编码到共享隐表征(sharedlatent represention,SLR),即中央节点中。这是受到人脑可以以不同的方式学习和表达抽象概念的启发。SLR的媒介不变属性可以被视为中间向量的高级正则化,强制它不仅捕获每个单个媒体内的隐式表示,如自动编码器,而且还可以像映射模型一样进行转换。此外,SLR模型是内容相关(content-specific)的,这意味着它只需要对数据集进行一次训练,同时用于不同的任务。我们展示了如何通过dropout训练SLR模型,并将其用于不同的序列到序列任务。我们的SLR模型在Youtube2Text和MSR-VTT数据集上得到验证,实现了视频到语句任务的显著效果提升,以及首次实现句子到视频的结果。
[Abstract]:In recent years, video data has been explosively growing. Such a large number of video data in the storage, identification, sharing, editing, generation and other processes need accurate video semantic analysis technology. At present, video semantic analysis based on depth learning can be divided into two steps: 1) extracting the visual feature expression of each frame by convolution neural network; 2) learning the feature sequence by using long-short term recurrent neural network (LSTM) and tabulating it. On the basis of a comprehensive survey and summary of existing video semantic analysis techniques, the classical problems in video semantic classification and video semantic description task depth learning models are fully studied. A continuous Dropout algorithm is proposed, which is a convolution neural network whose parameters are robust to image transformation and a convolution neural network whose structure is robust to image transformation. To solve this problem, an unsupervised layer-by-layer greedy learning approach is proposed to improve the model performance and training efficiency. Furthermore, in view of the limitations of the existing one-way mapping framework from video sequences to word sequences, a novel multi-way sequence learning algorithm based on latent semantic representation is proposed creatively. The main work and innovations of this paper are summarized as follows: Continuous Dropout Dropout has been proved to be an effective algorithm for training deep convolutional neural networks. Its main idea is that by shielding some atoms in a large-scale convolutional neural network, it can train more than one atom at a time. Enlightened by this phenomenon, we extend the traditional binary Dropout to continuous Dropout. On the one hand, continuous Dropout is closer to the activation of neurons in the human brain than traditional binary Dropout. On the other hand, we show that continuous Dropout has the property of avoiding the common adaptation of feature detectors. Results. The convolution neural network (CNN) with robust parameters has achieved the best results in many visual tasks. At present, almost all visual information is processed by convolution neural network. However, the current CNN model still shows poor robustness in image spatial transformation. The layered and parametric convolution neural networks based on the combination of convolution (matrix multiplication and nonlinear activation) and pool operation should be able to learn robust mapping from transformed input images to transformed invariant representations. On the contrary, each convolution kernel will learn invariant features in a variety of combinations of transformations for its input feature graph. Thus, we do not need to add any additional supervisory information to the optimization process and training image or modify the input image. CNN learning by machine transformation is more insensitive to the transformation of the input image. In small-scale image recognition, large-scale image recognition and image retrieval, the performance of the existing convolution neural network is improved. The robust convolution neural network convolution neural network (CNN) has shown the best performance in many visual recognition tasks. However, the combination of convolution and pooling operations shows little invariance to the local location of meaningful targets in the input. Sometimes, some networks use data augmentation to train the network to encode this invariance into network parameters, but this limits the ability of the model to learn the target content. In order to make the model concentrate on learning the object it describes, and not be affected by its position, we propose sorting the local blocks in the feature response graph, and then input them in. In the next layer, when block reordering combines convolution and pool operations, we obtain a consistent representation of targets in input images at different locations. We demonstrate that the proposed block reordering module can improve the performance of CNN for many benchmark tasks, including MNIST digital recognition, large-scale image recognition and image retrieval. Recent developments in sequential deep recurrent neural networks learning recurrent neural networks (RNNs), especially the long-and short-term memory networks (LSTMs) commonly used in video analysis, have shown their potential for modeling sequential data, especially in the areas of computer vision and natural language processing. Compared with the shallow network, the effect is not improved and the convergence speed is slow. This difficulty arises from the LSTM initialization method, in which the gradient-based optimization usually converges to the worse local solution. In this paper, we propose a novel encoder-decoder-based learning framework to initialize multi-layer LSTM in a greedy layer-by-layer training manner, in which each new LSTM layer is trained to retain the main information from the upper layer. Practicing multi-layer LSTM outperforms randomly initialized LSTM in terms of regression (additive problem), handwritten numeral recognition (MNIST), video classification (UCF-101) and machine translation (WMT'14). In addition, using greedy layer-by-layer training method, the convergence speed of multi-layer LSTM is increased by four times. Sequence-to-sequence learning sequence learning is a popular area of in-depth learning, such as video caption and speech recognition. Existing methods model the learning process by first encoding the input sequence into a fixed-size vector and then decoding the target sequence from the vector. Although simple and intuitive, this mapping model is task-dependent. In this paper, we propose a star-like framework for generic and flexible sequence-to-sequence learning in which different types of media content (peripheral nodes) can be encoded into shared latent representations (SLRs), or central nodes. The media-invariant properties of SLR can be viewed as high-level regularization of intermediate vectors, forcing it not only to capture implicit representations within each single medium, such as automatic encoders, but also to transform as a mapping model. In addition, the SLR model is content-specific. Our SLR model was validated on YouTube2Text and MSR-VTT datasets to achieve significant results for video-to-statement tasks. Upgrade, and first achieve sentence to video results.
【学位授予单位】：中国科学技术大学
【学位级别】：博士
【学位授予年份】：2017
【分类号】：TP391.41

【相似文献】