神经网络的权值规范化研究

发布时间：2018-02-23 20:16

本文关键词： 神经网络权值规范过拟合　出处：《吉林大学》2017年硕士论文　论文类型：学位论文

【摘要】：人工神经网络(Artificial Neural Network)通常又被称为神经网络,是将多个信息处理单元相互连接模拟生物学中细胞对信息的处理过程,该模型起源于1943年,由沃伦麦卡洛克和逻辑学家沃尔特共同提出,但在当时并未引起学术界的广泛关注。60年代当时神经网络的研究人员以及研究成果少的可怜。90年代由于深层网络梯度消失问题的发现和CNN反向传播问题的解决,神经网络曾有过短暂的繁荣。而现今,神经网络以及其演化的深度学习领域无疑是当下最热的课题之一,数以万计的科研工作者以及工程人员正投身于其中,神经网络又重新回到人们的视野中,这都得益于神经网络在图片识别,人工智能,商业推荐方面的卓越表现。神经网络同其他机器学习算法一样,在训练中存在过拟合问题,过拟合现象产生的原因主要是训练样本过少,导致网络并没有学习到样本的整体分布规律,从另一角度来看说明网络模型过于复杂,拟合了数据集中的噪声数据,而不是只从正确的数据进行学习。针对该问题不少学者,提出了一些防止网络过拟合的技术,其中应用最为广泛的要数规范化技术。神经网络的权值规范化通过增加对权重的惩罚相,防止在有限的训练数据下产生过于复杂的网络,能够有效地在非人工干预的情形下减弱模型的过拟合现象。寻找一个合适有效地规范化方法,对于网络的泛化效果有着重要的意义。当前神经网络规范化方法主要有两类,一类是基于权重空间发散程度的方法。例如L2规范化方法是寻找欧几里得空间内最快衰减方向,而L1规范化方法是在线性空间内寻找最快的衰减方向。更普遍的讲,当目标函数是凸函数时,网络的收敛速度与规范化方法相一致。L2方法约束的网络,其网络权重的收敛速度是二次非线性的,L1方法约束的网络,其权重的收敛是线性方式。另一种应用较广的权值规范化方法是dropout技术。这种约束方法并不采用显示的权重惩罚相,而是采用更加积极的方式,直接将某些权重连接抹去,从而抑制网络的过拟合。无论是现在广泛应用的L1还是L2规范化方法,从其定义形式中可以看出,他们都是基于网络全局的权重约束,L2规范化是在权重向量每个分量平方和之后求平方根,当限制L2规范化结果很小时,每一个权值都会很小。同样L1也存在类似的情况。不同的是,在L1约束下部分权值会为0。若能有一种有效基于局部的权值规范化方法。会对神经网络的研究有着重要的推动作用。本文深入探讨了影响神经网络性能的诸多指标,包括权重的初始化方法,损失函数的形式,激活神经元的类型。基于不平衡网络的特点,和线性神经元调节特性,着力提出一种基于传播路径的局部规范化方法,一个神经元的权重规范化值,取决于与其相连接的前向节点代价,和后一层相连接的节点代价值,基于线性神经元比例调节不变的性质,在L2规范化思想的基础上,加入了局部惩罚的规范化权值更新。本文实验部分在MINST数据集上分别对网络无规范化时、采用L1规范化方法、L2规范化方法、弃权技术方法时给出了对比结果。实验结果显示本文提出的基于传播路径的局部权值规范化算法,对防止网络过拟合,提高分类准确具有积极的意义。
[Abstract]:Artificial neural network (Artificial Neural Network) is also known as neural network, is a plurality of information processing units connected simulation process in biology of cell information, the model originated in 1943, jointly proposed by Warren MacCulloch and logician Walter, but at that time did not cause widespread concern in the academic researchers at.60 the neural network and the research results of little.90's as a result of the deep web to solve the problem of finding and gradient disappeared CNN back-propagation neural network problems, there had been a boom. However, deep learning neural network field and its evolution is undoubtedly one of the most hot topic at the moment, tens of thousands of scientists and engineers is engaged in the neural network back to people's vision, this is due to the neural network in image recognition, Artificial intelligence, business excellence. The recommended neural network with other machine learning algorithm, has the overfitting problem in training, causes the over fitting phenomenon is mainly the training sample is too small, and did not learn to lead the network distribution of the whole sample, from another point of view that the network model is too complex, fitting the noise data in data set, not only from the correct data for the study. Many scholars put forward some problems, to prevent over fitting network technology, which is most widely applied to the number of standardized technology. The weights of neural network standardization by increasing the weight of punishment, to prevent the generation of complex network when the training data is limited, can effectively reduce the phenomenon of over fitting model in non manual intervention situations. Looking for a suitable effective standardization method for Is of great significance to the network generalization effect. The neural network standardization method mainly has two kinds, one is the method of weight space based on the degree of divergence. For example the L2 standard method is to find the fastest in the Euclidean space attenuation direction, while the L1 standard method is to find the direction of attenuation of the fastest in the linear space. More generally, when the objective function is a convex function, the convergence speed and the standardized method of network.L2 is consistent in the constrained network, the network convergence speed is two times the weight of the nonlinear L1 method, constrained network convergence, its weight is linear. The weight normalization method for another application Guangzhou is dropout technology. This method is not constrained by weight penalty shows, but with a more positive way, directly to the certain weight connection erased, thereby inhibiting the over fitting of the network. It is now widely The application of the L1 or L2 standard method, from the definition of the form can be seen, they are the weight constraint network based on global L2 standard is the square root of the weight vector of each component square and after standardized results when restricted to L2 hours, each weight will be very small. Also L1 there is a similar situation. The difference is, under L1 constraint for 0. part weight if there is an effective method based on local standard weights. The study of neural network plays an important role. This paper discusses the influence of many indicators of the performance of the neural network, including the initialization method of weight loss the form of the function, types of activated neurons. Based on the unbalanced network characteristics, and linear neural regulation characteristics, puts forward a method based on the local standard propagation path, a neuron's weight normalized value, depending on the Connected to the first node cost, and connected after a layer node generation value, adjust the proportion of invariant linear neurons based on the nature, based on L2 standard idea, adding weights specification updates. Local punishment in the part of experiment on MINST data set is divided into no standardization of network L1, the standard method, L2 standard method, technology and methods are given waiver results. Experimental results show that the proposed algorithm based on local weights standard propagation path, to prevent network overfitting, is of positive significance to improve the classification accuracy.

【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP18

【相似文献】