Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes

This repository is an official implementation of the technical report AD-MLP.

Jiang-Tian Zhai*, Feng Ze*, Jinhao Du*, Yongqiang Mao*, Jiang-Jiang Liu†, Zichang Tan, Yifu Zhang, Xiaoqing Ye, Jingdong Wang†

Baidu Inc.

*: equal contribution, ^†: corresponding author.

News

• 2023.05.18: Paper is released on arxiv!
• 2023.05.17: Code / Models are released!

Introduction

Interesting Tips!

• We use a simple MLP-based approach that takes raw sensor data as input and outputs the future trajectory planning of the ego vehicle, without using any perception or prediction information such as camera images or LiDAR. This simple method achieves state-of-the-art end-to-end planning performance on the nuScenes Dataset, reducing the average L2 error by about 30% on the nuScenes open-loop evaluation metrics.
• The primary objective of this work is to present our observations rather than to propose a new method.
• Our findings demonstrate the potential limitations of the current evaluation scheme on the nuScenes dataset.
• Although our model performs well within the confines of the nuScenes Dataset, we acknowledge that it is merely an impractical toy incapable of functioning in real-world scenarios. Driving without any perception surrounding knowledge beyond the ego vehicle’s states is an insurmountable challenge.
• We hope our findings will stimulate further research in the field, encouraging a re-evaluation and enhancement of the planning task for end-to-end autonomous driving.

Results

• Open-loop planning results on nuScenes.

Method	L2 (m) 1s $\downarrow$	L2 (m) 2s $\downarrow$	L2 (m) 3s $\downarrow$	Col. (%) 1s $\downarrow$	Col. (%) 2s $\downarrow$	Col. (%) 3s $\downarrow$
ST-P3	1.33	2.11	2.90	0.23	0.62	1.27
UniAD	0.48	0.96	1.65	0.05	0.17	0.71
VAD-Tiny	0.20	0.38	0.65	0.10	0.12	0.27
VAD-Base	0.17	0.34	0.60	0.07	0.10	0.24
Ours	0.14	0.10	0.41	0.10	0.10	0.17

Get Started

• Environment Linux, Python==3.7.9, CUDA == 11.2, pytorch == 1.9.1 or paddlepaddle == 2.3.2. Besides, follow instruction in ST-P3 for running its evaluation process.

  cd deps/stp3
  conda env create -f environment.yml
  

• Prepare Data
  Download the nuScenes Dataset.

• Pretrained weights
  To verify the performance on the nuScenes Dataset, we provide the pretrained model weights (Google Drive and Baidu Netdisk). Please download them (paddle checkpoint, token of validation set...) to the root directory of this project.

• Paddle Evaluation

  python paddle/model/AD-MLP.py
  python deps/stp3/evaluate_for_mlp.py
  

  The first line saves the predicted 6 frames' trajectories of the next 3s in output_data.pkl. And the second line applies the ST-P3 evaluation on it. The final evaluation output contains the L2 error and collision rate in the next 1, 2 and 3s.

  Two versions of evaluation metrics are provided: online and offline. The offline version uses pre-stored ground truth and is far faster than online one. The code defaults to offline.

Contact

If you have any questions or suggestions about this repo, please feel free to contact us ([email protected], [email protected], [email protected], [email protected]).

Acknowledgement

This repo is build based on ST-P3. Thanks for their great work.

License

All code in this repository is under the Apache License 2.0.

BibTeX

If you find our work and this repository useful. Please consider giving a star and citation.

@article{zhai2023ADMLP,
  title={Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes},
  author={Zhai, Jiang-Tian and Feng, Ze and Du, Jihao and Mao, Yongqiang and Liu, Jiang-Jiang and Tan, Zichang and Zhang, Yifu and Ye, Xiaoqing and Wang, Jingdong},
  journal={arXiv preprint arXiv:2305.10430},
  year={2023}
}