Table of Contents
This repository contains the results of my bachelors thesis on the application of reinforcement learning in logistics distribution networks and inventory management. The project was implemented in python. The focus of this project was on the creation of an openAI Gym reinforcement learning environment containing a distribution network simulation. The implementation of different features was done in an iterative approach with frequent testing of the environment through common reinforcement learning agents from Stable-Baselines.
As of July 2022, because of current library incompatibilities, python version 3.7 needs to be installed.
- Install the base Gym library with
pip install gym - Install stable-baselines with
pip install stable-baselines3 - For compatibility reasons, an old version of tensorflow needs to be installed
pip install tensorflow==1.15.0 - To display graphs in the main file, install matplotlib
pip install matplotlib
If problems occur, compare all library versions to the collection below:
All Libraries/Versions used
- absl-py 1.1.0
- ale-py 0.7.5
- argon2-cffi 21.3.0
- argon2-cffi-bindings 21.2.0
- astor 0.8.1
- attrs 21.4.0
- backcall 0.2.0
- beautifulsoup4 4.11.1
- bleach 5.0.1
- cffi 1.15.1
- cloudpickle 2.1.0
- colorama 0.4.5
- cycler 0.11.0
- debugpy 1.6.0
- decorator 5.1.1
- defusedxml 0.7.1
- EditorConfig 0.12.3
- entrypoints 0.4
- fastjsonschema 2.15.3
- fonttools 4.33.3
- gast 0.2.2
- google-pasta 0.2.0
- grpcio 1.47.0
- gym 0.21.0
- gym-notices 0.0.7
- h5py 3.7.0
- importlib-metadata 4.12.0
- importlib-resources 5.8.0
- ipykernel 6.15.0
- ipython 7.34.0
- ipython-genutils 0.2.0
- jedi 0.18.1
- Jinja2 3.1.2
- joblib 1.1.0
- jsbeautifier 1.14.4
- jsonschema 4.6.1
- jupyter-client 7.3.4
- jupyter-core 4.10.0
- jupyterlab-pygments 0.2.2
- Keras-Applications 1.0.8
- Keras-Preprocessing 1.1.2
- kiwisolver 1.4.3
- lxml 4.9.1
- Markdown 3.3.7
- MarkupSafe 2.1.1
- matplotlib 3.5.2
- matplotlib-inline 0.1.3
- mistune 0.8.4
- nbclient 0.6.6
- nbconvert 6.5.0
- nbformat 5.4.0
- nest-asyncio 1.5.5
- notebook 6.4.12
- numpy 1.21.6
- opencv-python 4.6.0.66
- opt-einsum 3.3.0
- packaging 21.3
- pandas 1.1.5
- pandocfilters 1.5.0
- parso 0.8.3
- pickleshare 0.7.5
- Pillow 9.1.1
- pip 22.1.2
- prometheus-client 0.14.1
- prompt-toolkit 3.0.30
- protobuf 3.20.1
- psutil 5.9.1
- pycparser 2.21
- pygame 2.1.0
- Pygments 2.12.0
- pyparsing 3.0.9
- pyrsistent 0.18.1
- python-dateutil 2.8.2
- pytz 2022.1
- pywin32 304
- pywinpty 2.0.5
- pyzmq 23.2.0
- scipy 1.7.3
- Send2Trash 1.8.0
- setuptools 57.0.0
- six 1.16.0
- soupsieve 2.3.2.post1
- stable-baselines 2.10.2
- stable-baselines3 1.5.0
- tensorboard 1.15.0
- tensorflow 1.15.0
- tensorflow-estimator 1.15.1
- termcolor 1.1.0
- terminado 0.15.0
- tinycss2 1.1.1
- torch 1.12.0
- tornado 6.2
- traitlets 5.3.0
- typing_extensions 4.2.0
- wcwidth 0.2.5
- webencodings 0.5.1
- Werkzeug 2.1.2
- wheel 0.36.2
- wrapt 1.14.1
- zipp 3.8.
If only the Environment is needed, the class can be found in the rl_environment.py file. It can be used as a standard openAI Gym environment as described in the official Gym documentation.
The main.ipynb file provides a scenario in which the environment is getting set up and trained with a stable baselines model, as well as the application of the trained agent on a distribution network simulation.
The repository contains multiple versions of the artifact. They are organized in folders beginning with the version number and ending with the newest implemented feature. Inside the version folder the structure is as follows:
└── 0_example_feature <- Version Folder
│
├── main.ipynb <- Main notebook to train RL algorithmns and run simulations
├── rl_environment.py <- RL environment
│
├── simulation <- Folder: Simulation
│ ├── simulation.py <- Main simulation class. Controls actor classes. Starts simualtions
│ │
│ └── actor_classes <- Folder: Contains all simulation actor classes
│ ├── class_warehouse <- Abstract Warehouse class and subclasses RegionalWarehouse and CentralWarehouse
│ ├── class_customer <- Customer class
│ ├── class_manufacturer <- Manufacturer class (Not available in all versions)
│
└── logfiles <- Folder: Contains created logfiles from main.ipynb (Not available in all versions)
├── logfile_date_time.json <- Logfile containing information about a training/simulation run
└── ...