Atari

Work In Progress: Crossed out items have been partially implemented.

Prioritised experience replay [1] persistent advantage learning [2] bootstrapped [3] dueling [4] double [5] deep recurrent [6] Q-network [7] for the Arcade Learning Environment [8]. Or PERPALB(triple-D)RQN for short...

Additional asynchronous agents [9]:

• One-step Sarsa
• One-step Q-learning
• N-step Q-learning
• Advantage actor-critic

Run th main.lua to run headless, or qlua main.lua to display the game. The main options are -game to choose the ROM (see the ROM directory for more details) and -mode as either train or eval. Can visualise saliency maps [10], optionally using guided [11] or "deconvnet" [12] backpropagation. Saliency map modes are applied at runtime so that they can be applied retrospectively to saved models.

To run experiments based on hyperparameters specified in the individual papers, use ./run.sh <paper> <game> <args>. <args> can be used to overwrite arguments specified earlier (in the script); for more details see the script itself. By default the code trains on a demo environment called Catch - use ./run.sh demo to run the demo with good default parameters. Note that this code uses CUDA by default if available, but the Catch network is small enough that it runs faster on CPU.

In training mode if you want to quit using Ctrl+C then this will be caught and you will be asked if you would like to save the agent. Note that for non-asynchronous agents the experience replay memory will be included, totalling ~7GB. The main script also automatically saves the weights of the best performing DQN (according to the average validation score).

In evaluation mode you can create recordings with -record true (requires FFmpeg); this does not require using qlua. Recordings will be stored in the videos directory.

Requirements

Requires Torch7, and uses CUDA if available. Also requires the following extra luarocks packages:

• luaposix
• moses
• logroll
• classic
• torchx
• rnn
• dpnn
• nninit
• tds
• xitari
• alewrap
• rlenvs

xitari, alewrap and rlenvs can be installed using the following commands:

luarocks install https://raw.githubusercontent.com/lake4790k/xitari/master/xitari-0-0.rockspec
luarocks install https://raw.githubusercontent.com/Kaixhin/alewrap/master/alewrap-0-0.rockspec
luarocks install https://raw.githubusercontent.com/Kaixhin/rlenvs/master/rocks/rlenvs-scm-1.rockspec

Acknowledgements

• @GeorgOstrovski for confirmation on network usage in advantage operators + note on interaction with Double DQN.
• @schaul for clarifications on prioritised experience replay + dueling DQN hyperparameters.

References

[1] Prioritized Experience Replay
[2] Increasing the Action Gap: New Operators for Reinforcement Learning
[3] Deep Exploration via Bootstrapped DQN
[4] Dueling Network Architectures for Deep Reinforcement Learning
[5] Deep Reinforcement Learning with Double Q-learning
[6] Deep Recurrent Q-Learning for Partially Observable MDPs
[7] Playing Atari with Deep Reinforcement Learning
[8] The Arcade Learning Environment: An Evaluation Platform for General Agents
[9] Asynchronous Methods for Deep Reinforcement Learning
[10] Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
[11] Striving for Simplicity: The All Convolutional Net
[12] Visualizing and Understanding Convolutional Networks