CppRl - PyTorch C++ Reinforcement Learning

Above: results on LunarLander-v2 after 60 seconds of training on my laptop

CppRl is a reinforcement learning framework, written using the PyTorch C++ frontend.

It is very heavily based on Ikostrikov's wonderful pytorch-a2c-ppo-acktr-gail. You could even consider this a port. The API and underlying algorithms are almost identical (with the necessary changes involved in the move to C++).

It also contains an implementation of a simple OpenAI Gym server that communicates via ZeroMQ to test the framework on Gym environments.

CppRl aims to be an extensible, reasonably optimized, production-ready framework for using reinforcement learning in projects where Python isn't viable. It should be ready to use in desktop applications on user's computers with minimal setup required on the user's side.

Motivation

At the time of writing, there are no general-use reinforcement learning frameworks for C++. I needed one for a personal project, and the PyTorch C++ frontend had recently been released, so I figured I should make one.

Features

• Implemented algorithms:
  • A2C
  • PPO
• Recurrent policies (GRU based)
• Cross-platform compatibility (tested on Windows 10, Ubuntu 16.04, and Ubuntu 18.04)
• Solid test coverage
• Decently optimized (always open to pull requests improving optimization though)

Example

An example that uses the included OpenAI Gym server is provided in example. It can be run as follows: Terminal 1:

./launch_gym_server.py

Terminal 2:

build/example/gym_server

It takes about 60 seconds to train an agent to 200 average reward on my laptop (i7-8550U processor).

The environment and hyperparameters can be set in example/gym_client.cpp.

Note: The Gym server and client aren't very well optimized, especially when it comes to environments with image observations. There are a few extra copies necessitated by using an inter-process communication system, and then gym_client.cpp has an extra copy or two to turn the observations into PyTorch tensors. This is why the performance isn't that good when compared with Python libraries running Gym environments.

Building

CMake is used for the build system. Most dependencies are included as submodules (run git submodule update --init --recursive to get them). Libtorch has to be installed seperately. The OpenAI Gym client also uses Zmpqpp, which can be installed with sudo apt-get install libzmqpp-dev.

cd pytorch-cpp-rl
mkdir build && cd build
cmake ..
make -j4

Windows build instructions coming soon.

Testing

You can run the tests with build/cpprl_tests.