simulation_benchmark

Benchmark comparison for rigid-body dynamic simulators. This project is part of GSoC'24@OpenRobotics.

This project is an effort to create an open-source benchmarking suite for robotics physics engines/simulators (for example: gazebo, mujoco, and drake).

Gazebo Classic benchmark is available here

• Boxes benchmark: Free-floating rigid bodies:

  

• Triball benchmark: Rigid bodies in contact:

Steps to run benchmark:

Install dependencies:

pip install lz4 protobuf zstandard matplotlib numpy pandas
sudo apt-get install python3-gz-math7

Install currently implemented simulators:

• Gazebo Ionic:

  Installation

  • Follow the Gazebo Ionic source installation up to here.
  • Edit collection-ionic.yaml.
  • In gz-sim version: main, replace main with scepters/set_model_state_prototype.

  This is how it should look:

  gz-sim:
  type: git
  url: https://github.com/gazebosim/gz-sim
  version: scpeters/set_model_state_prototype

  Follow rest of the installation tutorial.

Build and run tests:

git clone https://github.com/yaswanth1701/simulation_benchmark.git
cd benchmark
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make 
make test 

Run Python notebooks for results

• boxes_results.ipynb

Benchmark suite feature description:

Benchmark components:

• Simulation & logging -> mcap to csv log conversion -> post_processing -> result_plots

This suite offers the following features:

• Logging (mcap & csv formats)
• Dynamics world generation at run-time(sdf.erb format).
• Simulator independent post-processing script.

These features allow for a simulator-independent suite for benchmarking.

Currently implemented simulators/physics engines:

• Gazebo Ionic benchmark:

  • DART
  • Bullet
  • Bullet-Featherstone

Future works:

• Benchmarks with the model having joints and multi-body dynamics.
• Integration of Mujoco and Drake.

Call for Contributions and Feedback:

We would highly appreciate developers of open-source simulators and the robotics community suggesting or implementing new benchmarking ideas using this benchmarking suite and opening a pull request for it. If anyone comes across problems while running the benchmark, please let us know by opening an issue.