Dopamine
Our design principles are:
- Easy experimentation: Make it easy for new users to run benchmark
- Flexible development: Make it easy for new users to try out research ideas.
- Compact and reliable: Provide implementations for a few, battle-tested
- Reproducible: Facilitate reproducibility in results. In particular, our
setup follows the recommendations given by
In the spirit of these principles, this first version focuses on supporting the
state-of-the-art, single-GPU Rainbow agent ()
applied to Atari 2600 game-playing (Bellemare et al., 2013).
Specifically, our Rainbow agent implements the three components identified as
most important by :
- n-step Bellman updates (see e.g. Mnih et al., 2016)
- Prioritized experience replay ()
- Distributional reinforcement learning (C51; Bellemare et al., 2017)
For completeness, we also provide an implementation of DQN
().
For additional details, please see our
documentation.
This is not an official Google product.
- 01/11/2018: Download links for each individual checkpoint, to avoid
having to download all of the checkpoints. - 29/10/2018: Graph definitions now show up in Tensorboard.
- 16/10/2018: Fixed a subtle bug in the IQN implementation and upated
the colab tools, the JSON files, and all the downloadable data. - 18/09/2018: Added support for double-DQN style updates for the
ImplicitQuantileAgent
.- Can be enabled via the
double_dqn
constructor parameter.
- Can be enabled via the
- 18/09/2018: Added support for reporting in-iteration losses directly from
the agent to Tensorboard.- Include the flag
--debug_mode
in your command line to enable it. - Control frequency of writes with the
summary_writing_frequency
agent constructor parameter (defaults to500
).
- Include the flag
- 27/08/2018: Dopamine launched!
Instructions
Installing from source allows you to modify the agents and experiments as
you please, and is likely to be the pathway of choice for long-term use.
These instructions assume that you’ve already set up your favourite package
manager (e.g. apt
on Ubuntu, homebrew
on Mac OS X), and that a C++ compiler
is available from the command-line (almost certainly the case if your favourite
package manager works).
The instructions below assume that you will be running Dopamine in a virtual
environment. A virtual environment lets you control which dependencies are
installed for which program; however, this step is optional and you may choose
to ignore it.
Dopamine is a Tensorflow-based framework, and we recommend you also consult
the
for additional details.
Finally, these instructions are for Python 2.7. While Dopamine is Python 3
compatible, there may be some additional steps needed during installation.
Ubuntu
First set up the virtual environment:
sudo apt-get update && sudo apt-get install virtualenv
virtualenv --python=python2.7 dopamine-env
source dopamine-env/bin/activate
This will create a directory called in which your virtual
environment lives. The last command activates the environment.
During installation, you may safely ignore the following error message:
tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you’ll have
numpy 1.15.1 which is incompatible.
Finally, download the Dopamine source, e.g.
git clone https://github.com/google/dopamine.git
Mac OS X
First set up the virtual environment:
virtualenv --python=python2.7 dopamine-env
source dopamine-env/bin/activate
This will create a directory called dopamine-env
in which your virtual
environment lives. The last command activates the environment.
Then, install the dependencies to Dopamine:
brew install cmake zlib
pip install absl-py atari-py gin-config gym opencv-python tensorflow
During installation, you may safely ignore the following error message:
tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you’ll have
numpy 1.15.1 which is incompatible.
Finally, download the Dopamine source, e.g.
Running tests
You can test whether the installation was successful by running the following:
cd dopamine
export PYTHONPATH=${PYTHONPATH}:.
python tests/atari_init_test.py
The entry point to the standard Atari 2600 experiment is
.
To run the basic DQN agent,
python -um dopamine.atari.train \
--agent_name=dqn \
--base_dir=/tmp/dopamine \
--gin_files='dopamine/agents/dqn/configs/dqn.gin'
By default, this will kick off an experiment lasting 200 million frames.
The command-line interface will output statistics about the latest training
episode:
[...]
I0824 17:13:33.078342 140196395337472 tf_logging.py:115] gamma: 0.990000
I0824 17:13:33.795608 140196395337472 tf_logging.py:115] Beginning training...
Steps executed: 5903 Episode length: 1203 Return: -19.
More generally, the whole of Dopamine is easily configured using the
.
An easy, alternative way to install Dopamine is as a Python library:
Depending on your particular system configuration, you may also need to install
zlib (see “Install via source” above).
Running tests
From the root directory, tests can be run with a command such as:
python -um tests.agents.rainbow.rainbow_agent_test
Marc G. Bellemare, Pablo Samuel Castro, Carles Gelada, Saurabh Kumar, Subhodeep Moitra.
Dopamine, https://github.com/google/dopamine, 2018.