连续空间中的一种动作加权行动者评论家算法

发布时间：2018-01-29 08:29

本文关键词： 强化学习连续空间函数逼近行动者评论家梯度下降人工智能　出处：《计算机学报》2017年06期 　论文类型：期刊论文

【摘要】：经典的强化学习算法主要应用于离散状态动作空间中.在复杂的学习环境下,离散空间的强化学习方法不能很好地满足实际需求,而常用的连续空间的方法最优策略的震荡幅度较大.针对连续空间下具有区间约束的连续动作空间的最优控制问题,提出了一种动作加权的行动者评论家算法(Action Weight Policy Search Actor Critic,AW-PS-AC).AW-PS-AC算法以行动者评论家为基本框架,对最优状态值函数和最优策略使用线性函数逼近器进行近似,通过梯度下降方法对一组值函数参数和两组策略参数进行更新.对两组策略参数进行加权获得最优策略,并对获得的最优动作通过区间进行约束,以防止动作越界.为了进一步提高算法的收敛速度,设计了一种改进的时间差分算法,即采用值函数的时间差分误差来更新最优策略,并引入了策略资格迹调整策略参数.为了证明算法的收敛性,在指定的假设条件下对AW-PS-AC算法的收敛性进行了分析.为了验证AW-PS-AC算法的有效性,在平衡杆和水洼世界实验中对AW-PS-AC算法进行仿真.实验结果表明AW-PS-AC算法在两个实验中均能有效求解连续空间中近似最优策略问题,并且与经典的连续动作空间算法相比,该算法具有收敛速度快和稳定性高的优点.
[Abstract]:The classical reinforcement learning algorithm is mainly used in discrete state action space. In the complex learning environment, the reinforcement learning method in discrete space can not meet the actual needs. However, the usual method of continuous space has a large amplitude of oscillation. The optimal control problem of continuous action space with interval constraints in continuous space is discussed. This paper presents an actor-weighted actor-critic algorithm named Action Weight Policy Search Actor Critic. The AW-PS-AC).AW-PS-AC algorithm takes the actor critic as the basic frame and approximates the optimal state value function and the optimal strategy using the linear function approximator. One set of value function parameters and two groups of policy parameters are updated by gradient descent method. The optimal strategy is obtained by weighting the two groups of policy parameters, and the obtained optimal actions are constrained through the interval. In order to prevent the action from crossing the boundary. In order to further improve the convergence speed of the algorithm, an improved time-difference division algorithm is designed, that is, the time-difference error of the value function is used to update the optimal strategy. The policy parameters are introduced to prove the convergence of the algorithm. The convergence of AW-PS-AC algorithm is analyzed under the specified assumptions. In order to verify the validity of AW-PS-AC algorithm. The AW-PS-AC algorithm is simulated in the balance bar and water pool world experiments. The experimental results show that the AW-PS-AC algorithm can effectively solve the approximate optimal strategy problem in the continuous space in both experiments. Compared with the classical continuous action space algorithm, this algorithm has the advantages of fast convergence and high stability.
【作者单位】：苏州大学计算机科学与技术学院;软件新技术与产业化协同创新中心;吉林大学符号计算与知识工程教育部重点实验室;
【基金】：国家自然科学基金(61472262,61502323,61502329) 江苏省自然科学基金(BK2012616) 江苏省高校自然科学研究项目(13KJB520020) 吉林大学符号计算与知识工程教育部重点实验室基金项目(93K172014K04) 苏州市应用基础研究计划工业部分(SYG201422,SYG201308)资助~~
【分类号】：TP18
【正文快照】： 金(BK2012616)、江苏省高校自然科学研究项目(13KJB520020)、吉林大学符号计算与知识工程教育部重点实验室基金项目(93K172014K04)、苏州市应用基础研究计划工业部分(SYG201422,SYG201308)资助.刘全,男,1969年生,博士,教授,博士生导师,中国计算机学会(CCF)高级会员,主要研究领

【相似文献】