一种基于视觉注意力机制的深度循环Q网络模型

发布时间：2018-04-18 16:08

本文选题：深度学习 + 强化学习　；参考：《计算机学报》2017年06期

【摘要】：由现代强化学习和深度学习相结合形成的深度强化学习方法是目前人工智能领域一个新的研究热点,已经在各种需要感知高维度原始输入数据和决策控制的任务中取得了实质性的突破.尤其是一种被称为深度Q网络的模型在处理诸如Atari 2600游戏这类趋于真实环境的复杂问题时表现出了和人类玩家相媲美的水平.然而,当存在有延迟的奖赏而导致需要长时间步规划才能优化策略的情形中,深度Q网络的表现就会急剧下降.这说明深度Q网络并不擅长解决战略性深度强化学习任务.针对此问题,文中使用带视觉注意力机制的循环神经网络改进了传统的深度Q网络模型,提出了一种较为完善的深度强化学习模型.新模型的关键思想有两点:一是使用双层门限循环单元构成的循环神经网络模块来记忆较长时间步内的历史信息.这使得Agent能够及时使用有延迟的反馈奖赏来正确地指导下一步的动作选择;二是通过视觉注意力机制自适应地将注意力集中于面积较小但更具价值的图像区域,从而使得Agent能够更加高效地学习近似最优策略.该文通过选取一些经典的Atari 2600战略性游戏作为实验对象来评估新模型的有效性.实验结果表明,与传统的深度强化学习模型相比,新模型在一些战略性任务上具有很好的性能表现和较高的稳定性.
[Abstract]:The method of depth reinforcement learning, which is formed by the combination of modern reinforcement learning and deep learning, is a new research hotspot in the field of artificial intelligence.A substantial breakthrough has been made in various tasks requiring high dimensional raw input data and decision control.In particular, a model called Deep Q Network has shown a level comparable to that of human players in dealing with complex problems such as Atari 2600 games that tend to be real-world.However, when there is a delayed reward, the performance of the deep Q network will decline sharply because of the need of long step planning in order to optimize the strategy.This shows that the depth Q network is not good at solving strategic intensive learning tasks.To solve this problem, the traditional depth Q network model is improved by using the circular neural network with visual attention mechanism, and a more perfect depth reinforcement learning model is proposed.There are two key ideas in the new model: one is to use the circulatory neural network module composed of double-layer threshold cyclic unit to memorize the historical information in a long time step.This allows Agent to use delayed feedback rewards in time to correctly guide the next step of action selection; second, it adaptively focuses on smaller but more valuable image areas through visual attention mechanisms.Thus, Agent can learn approximate optimal strategy more efficiently.This paper evaluates the effectiveness of the new model by selecting some classic Atari 2600 strategic games as experimental objects.The experimental results show that the new model has good performance and stability in some strategic tasks compared with the traditional depth reinforcement learning model.
【作者单位】：苏州大学计算机科学与技术学院;软件新技术与产业化协同创新中心;吉林大学符号计算与知识工程教育部重点实验室;
【基金】：国家自然科学基金项目(61272005,61303108,61373094,61472262,61502323,61502329) 江苏省自然科学基金(BK2012616) 江苏省高校自然科学研究项目(13KJB520020,16KJB520041) 吉林大学符号计算与知识工程教育部重点实验室基金项目(93K172014K04) 苏州市应用基础研究计划工业部分(SYG201422,SYG201308)资助~~
【分类号】：TP18

【参考文献】