DOMUS gym environment for ml-sim

Based on https://www.gymlibrary.ml/content/environment_creation/

Installation and running

Make sure you have miniconda or anaconda installed in the usual way - https://docs.conda.io/en/latest/miniconda.html

Install pytorch with GPU or CPU support - https://pytorch.org/get-started/locally/

Create a conda environment or alternatively create an appropriate virtual environment

conda create -n domusgym 
conda activate domusgym

or

virtualenv venv
source venv/bin/activate

Note that it may also be necessary to install gfortran as this is needed to install scipy

sudo apt install gfortran

Install domus_mlsim and domus_gym

git clone --recurse-submodules https://github.com/jbrusey/domus_gym.git
(cd domus_gym/domus_mlsim && pip install -e .)
(cd domus_gym && pip install -e .)

Make sure you have an adequate build environment. For Linux this means installing build-essential along with some other things required by rl-zoo.

sudo apt install build-essential cmake swig ffmpeg zlib1g-dev

Install rl-zoo for stable-baselines3

git clone https://github.com/DLR-RM/rl-baselines3-zoo.git
pip install -r rl-baselines3-zoo/requirements.txt

Add the following into rl-baselines3-zoo/hyperparams/ppo.yml

cat >>rl-baselines3-zoo/hyperparams/ppo.yml <<EOF
Domus-v0:
  normalize: true
  n_envs: 8
  n_timesteps: !!float 2e6
  policy: 'MlpPolicy'
  n_steps: 2048
  batch_size: 64
  gae_lambda: 0.95
  gamma: 0.99
  n_epochs: 10
  ent_coef: 0.0
  learning_rate: 2.5e-4
  clip_range: 0.2

DomusCont-v0:
  normalize: true
  n_envs: 8
  n_timesteps: !!float 2e6
  policy: 'MlpPolicy'
  n_steps: 2048
  batch_size: 64
  gae_lambda: 0.95
  gamma: 0.99
  n_epochs: 10
  ent_coef: 0.0
  learning_rate: 2.5e-4
  clip_range: 0.2

DomusFullAct-v0:
  normalize: true
  n_envs: 8
  n_timesteps: !!float 2e7
  policy: 'MlpPolicy'
  n_steps: 2048
  batch_size: 64
  gae_lambda: 0.95
  gamma: 0.99
  n_epochs: 10
  ent_coef: 0.0
  learning_rate: 2.5e-4
  clip_range: 0.2
EOF

Train the agent (this will take around 24 hours on a fast machine).

python train.py --algo ppo --env DomusFullAct-v0 --gym-packages domus_gym --env-kwargs use_random_scenario:True --verbose 1