回声状态网络的结构参数优化及其应用

发布时间：2018-01-01 11:23

本文关键词：回声状态网络的结构参数优化及其应用　出处：《华东理工大学》2016年博士论文　论文类型：学位论文

【摘要】：近年来,储存池计算(Reservoir Computing, RC)发展非常迅速,并且已经成功应用于很多领域。回声状态网络(Echo State Network, ESN),液体状态机(Liquid State Machines, LSMs)及BP解相关(Back-propagation Decorrelation, BPDC)是最常见的RC方法。ESN是递归神经网络(Recurrent Neural Network, RNN)的一种,它由不需要训练的稀疏连接隐层和需要训练确定的线性输出层组成,通常情况下,ESN的输入权值、储存池拓扑结构及权值都是随机初始化的。ESN已经成功应用于时间序列预测、非线性系统辨识、语言处理、手机流量预测、股票预测及语言模型等领域,在暂态时序系统建模上表现出很强的潜能和前景。但同时ESN也有着很多缺点,例如：难以理解的储存池黑箱特性、需要多次尝试才能确定的储存池连接结构和权值、缺少原则性的储存池创建以及没有原则的设计框架。随机初始化的储存池很难达到最优,同时储存池的黑箱结构也很难使我们清晰地认识储存池的动态特性。所以针对特定对象如何确定ESN输入缩放系数、最优储存池拓扑结构以及最优输出层连接等,仍然是ESN研究的热点。本文的研究内容主要分为ESN的非储存池结构和参数的优化(输入缩放参数的调节和输出层连接的结构优化)和储存池优化(储存池大小和权值调整)两个层次。本文的主要研究成果如下：(1)在多输入ESN建模中,存在输入缩放系数调节的问题,目前,在多输入的条件下,很少有人研究ESN的输入缩放系数和输出层连接结构对网络性能的影响,同时也缺少一种公认的ESN理论设计框架。针对ESN一些难以确定的参数如：输出层结构、输入缩放参数、谱半径调节等问题,本文提出了基于输入缩放系数调节(Sensitivity Analysis Input ScalingRegulation, S AISR)和冗余信息剪枝(Redundant Unit Pruning Algorithm, RUPA) 的 ESN模型。在SAISR中,首先通过默认输入缩放系数创建并初始化一个足够大的ESN,然后训练已建立好的ESN并计算每个输入变量的敏感度,最后根据敏感度的大小得到ESN的最优输入缩放系数,并用最优输入缩放系数重新建立ESN模型。在RUPA中,首先计算各个储存池神经元之间的相关系数从而找到最相关的两个储存池神经元,最相关的神经元呈冗余关系,然后剪掉其中一个冗余神经元的输出连接,同时重复循环剪枝过程直到误差达到期望值以内。最终通过对青霉素批处理过程的建模发现在多输入的条件下,输入缩放系数比ESN的其他参数如：储存池谱半径、稀疏度及储存池缩放因子对网络性能的影响大。同时RUPA可以在提升网络性能的情况下进一步简化网络。(2)ESN输出连接的优化是一个特征子集选取问题,对ESN的输出层连接进行优化可以得到更好的输出层连接结构,输出连接优化的本质是一个离散、高维、非线性的优化问题。为了进一步简化网络和提升网络的泛化性,提出了一种基于二进制粒子群优化输出连接的回声状态网络,由于输出层连接的优化是一个二进制的特征提取问题,所以本文采用二进制粒子群(Binary Particle Swarm Optimization, BPSO)来解决输出连接优化问题,首先根据训练数据建立一个足够大的ESN,同时训练已建立的ESN,计算其误差函数并产生初始化种群,然后根据每个粒子的适应值(验证误差)来更新种群,最终通过得到的最优粒子,也就是最优输出层连接来建立优化模型。将所提方法应用于三个时序对象和青霉素批处理过程,仿真结果表明了全局优化方法BPSO与相关性剪枝方法RUPA、传统特征子集选取方法(Least Angle Regression, LAR)相比,BPSO更能有效的优化输出层连接,同时在简化程度上,全局优化方法也优于RUPA和LAR。(3)由于ESN的储存池初始化是随机的,所以储存池很难达到最优,同时储存池的随机性导致了不清晰的储存池的动态特性,如何设计和构建储存池仍是ESN研究的难点。确定型储存池拓扑结构ESN的出现大大改进了传统ESN,消除了传统ESN的随机性,并且性能上不亚于传统ESN。但是如何确定其储存池的大小仍然是一个没有解决的问题,针对如何确定SCR(Simple Cycle Reservoir, SCR)型储存池ESN (SCRN)的大小提出了基于剪枝或构造改进的确定型储存池回声状态网络,利用构造法或剪枝法来确定SCRN的储存池大小。构造法采用简单分裂算法(Simple Disjunction Algorithm, SDA),首先构建一个合适大小的SCRN,然后计算每个储存池神经元的权值权重,最后根据计算得来的储存池权值权重,把超过一定阀值的神经元分裂为两个。而剪枝算法采用敏感迭代剪枝算法(Sensitive Iterated Pruning Algorithm, SIPA),首先构建一个相对过大的SCRN,然后计算每个储存池神经元的敏感度,剪掉敏感度最低的储存池神经元,最后通过最小二乘法调整剩余神经元的权值从而保持储存池信息的完整性。仿真结果证实,对于SCRN,剪枝方法可以提升网络的泛化性能并简化网络,剪枝方法与构造方法相比,剪枝方法更适合用来确定SCRN的储存池大小。虽然SDA的性能不及剪枝方法,但是SDA可以明显的减少输出层的权值系数,同时可以略微提高SCRN的稳定性,由实验结果可以看出,从提升性能上来讲,剪枝方法对ESN'性能的提升要远远高于构造法；从稳定性上来讲,构造法更能提升ESN的稳定性。
[Abstract]:In recent years, storage pool calculation (Reservoir Computing, RC) is developing very rapidly, and has been successfully used in many fields. The echo state network (Echo State Network, ESN), liquid state machine (Liquid State Machines, LSMs) and BP (Back-propagation Decorrelation, BPDC solution.ESN RC) is the most common method is recursive neural network (Recurrent Neural Network, RNN) a, it does not require training by sparse connection of the hidden layer and the output layer to determine the training needs of linear composition, usually, the input weight ESN, storage pool topology and weights are randomly initialized.ESN has been successfully applied to nonlinear time series prediction. System identification, language processing, mobile phone traffic forecast, stock prediction and language model and other fields, show the potential and Prospect of the strong modeling transient timing system. But at the same time, ESN also have many shortcomings, For example: the storage pool is difficult to understand the characteristics of the black box, take several attempts to determine the storage pool connection structure and weights, lack of principle and create a storage pool design framework without principle. The random initialization of the storage pool is difficult to achieve the optimal storage at the same time, the dynamic characteristics of the black box structure pool is also difficult to make a clear understanding of our store for a specific object pool. So how to determine the ESN input zoom coefficient, the optimal storage pool topology and the optimal output layer connection, is still the hotspot of ESN research. The optimization of the main content of this paper is divided into the storage tank structure and the parameters of the ESN (input and output parameters of the zoom adjustment layer connection structure optimization) (storage and storage pool optimization adjustment pool size and weight) two levels. The main results are as follows: (1) in multi input ESN modeling, input zoom adjustment coefficient At present, the problems in multi input conditions, have little effect on the input of the ESN zoom coefficient and output layer connection structure on the performance of the network, but also the lack of a recognized ESN design theory framework. Using ESN to determine the parameters such as the output layer structure, input zoom parameters, adjusting the spectrum radius, is proposed to adjust the input coefficient based on zoom (Sensitivity Analysis Input ScalingRegulation S AISR (Redundant) and Unit Pruning redundancy pruning Algorithm, RUPA) ESN model. In SAISR, the default input zoom coefficient to create and initialize a large enough ESN, then established ESN and training the calculation of each input variable sensitivity, according to the sensitivity of the size of the optimal input zoom coefficient ESN, and the optimal input coefficient to establish the ESN model zoom in RUPA. First of all, calculate the correlation coefficient of each storage pool between neurons in order to find the most relevant two storage pool is the most relevant relationship between neurons, redundant neurons, and then cut the output of a redundant neuron connection, and repeat the cycle pruning process until the error reaches the expected value within. By the end of penicillin batch process modeling found in under the condition of multi input, input the zoom factor than other parameters such as ESN storage pool spectral radius, sparsity and storage pool zoom factor on the performance of the network. At the same time, RUPA can further simplify the network to enhance the performance of the network under the situation. (2) the ESN output is connected to the optimization problem of selecting a subset of features the output layer of ESN, connection optimization can get better connected to the output layer structure, the nature of the output connection optimization is a discrete, high dimension, nonlinear optimization problems In order to further simplify the generalization problem. Network and improve the network, put forward a kind of echo state network binary particle swarm optimization based on output connection, because the output layer connection optimization is a binary feature extraction problem, so this paper uses binary particle swarm (Binary Particle Swarm Optimization, BPSO) to solve the optimization problem of the output connection first of all, according to the training data to build a large enough ESN, while training the established ESN, calculate the error function and population initialization, then according to the fitness value of each particle (verification error) to update the population, finally obtained the optimal particle, which is the optimal output layer connection optimization model is established. The proposed method is applied to three sequential objects and penicillin batch process, the simulation results show that the global optimization method of BPSO and RUPA correlation pruning method, traditional. Subset selection method (Least Angle Regression, LAR) compared to BPSO can optimize the output layer effective connection, while simplifying extent is better than that of RUPA and LAR. method for global optimization (3) due to the ESN storage pool initialization is random, so it is difficult to achieve the optimal storage pool, and random storage pool the resulting dynamic characteristics is not clear the storage pool, how to design and build a ESN storage pool is still difficult. The type of storage tank ESN topology greatly improves the traditional ESN, eliminate the randomness of traditional ESN, and the performance is not less than the traditional ESN. but how to determine the storage pool size is still an unresolved issue, to determine the SCR (Simple Cycle Reservoir, SCR) ESN (SCRN) type storage pool size is proposed to determine the type of structure based on improved pruning or storage pool echo state network, using construction method Or pruning method to determine the SCRN pool size. A simple split algorithm based on the construction method (Simple Disjunction Algorithm, SDA), the first to build a suitable size of SCRN, and then calculate the weight weight of each storage pool of neurons, the final weight according to the calculated to the storage pool right value exceeds a certain threshold to split into neurons two. The pruning algorithm using sensitive iterative pruning algorithm (Sensitive Iterated Pruning Algorithm, SIPA), first built a relatively large SCRN, and then calculated for each storage pool of neurons sensitive to cut storage pool the least sensitive neurons, finally through the least squares method to adjust the weights of the remaining neurons to maintain the integrity of the store the pool of information. The simulation results show that for SCRN, the pruning method can improve the generalization performance of the network and simplify the network structure, compared with the method of pruning, pruning The method is more suitable to determine the SCRN pool size. Although the performance of SDA is less than the pruning method, but the SDA can significantly reduce the weight coefficient of the output layer, and can slightly improve the stability of the SCRN, can be seen from the experimental results from the performance of speaking, the pruning method improve the performance of ESN'is much higher than the construction method from the stability; speaking, construction method can enhance the stability of the ESN.

【学位授予单位】：华东理工大学
【学位级别】：博士
【学位授予年份】：2016
【分类号】：TP18

【相似文献】