SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

Changhui Jing, Haochen Tian, Hongchen Li, Tianyu Li, Caojun Wang, Zihao Zhang, Yanhao Liu, Chengen Xie, Bailin Li, Hao Zhao, Zhenjie Yang

• 📧 Primary Contact: Changhui Jing (changhui.jing@opendrivelab.com)
• 🖊️ Joint effort by Shanghai Innovation Institute, OpenDriveLab at HKU, Li Auto Inc., Tongji University, and Tsinghua University.

---

Highlights

• 🚗 Simulation-grounded alignment: Render matched future videos for alternative candidate trajectories in reconstructed real-world scenes.
• 🔍 Selective video supervision: Prioritize higher-error futures among high-reward candidates and jointly train trajectory planning and video prediction.
• 📈 Improved planning: Achieve 91.3 PDMS on NAVSIM v1, 88.2 EPDMS on NAVSIM v2, and 38.8 average HD-Score on HUGSIM, with direct trajectory prediction at inference.

News

• [2026/10/23] We released the checkpoints and inference code of SimVTA.
• [2026/10/12] We released our paper on arXiv.

📋 TODO List

• Results release.
• Checkpoint release.
• Inference and evaluation code.
• Training code.

Getting Started

1. Clone SimVTA

git clone https://github.com/OpenDriveLab/SimVTA.git
cd SimVTA

2. Create Environment

conda create -n simvta python=3.10 -y
conda activate simvta
pip install -r requirements.txt
pip install -e .

Use a CUDA-compatible PyTorch installation and a GPU with bfloat16 support. NAVSIM v1 is included in navsim/.

3. Data Preparation

Prepare the NAVSIM v1 OpenScene navtest logs, sensor blobs, and nuPlan maps following the NAVSIM data instructions. Organize the files as follows:

SimVTA/
├── checkpoints/
│   ├── simvta.pkl
│   └── dcae_td_20000.pkl
├── datasets/
│   ├── maps/
│   │   └── nuplan-maps-v1.0/
│   └── openscene/
│       ├── navsim_logs/
│       │   └── test/
│       └── sensor_blobs/
└── exp/
    └── metric_cache/

4. Download Checkpoints

Download simvta.pkl from OpenDriveLab-org/SimVTA and the temporal-aware DCAE checkpoint dcae_td_20000.pkl from Kevin-thu/Epona. Place both files in checkpoints/:

mkdir -p checkpoints
wget -c https://huggingface.co/OpenDriveLab-org/SimVTA/resolve/main/simvta.pkl -O checkpoints/simvta.pkl
wget -c https://huggingface.co/Kevin-thu/Epona/resolve/main/dcae_td_20000.pkl -O checkpoints/dcae_td_20000.pkl

The DCAE checkpoint encodes the input camera images into latents.

5. NAVSIM Evaluation

bash run/test_navsim.sh

Overview

SimVTA is a simulation-grounded post-training framework for driving world action models. It extends supervision beyond logged trajectories using a 3D Gaussian Splatting simulator, combining trajectory GRPO with a flow-matching video loss that updates shared representations.

Reported results in the paper:

Benchmark	Metric	SimVTA
NAVSIM v1	PDMS	91.3 (+5.1 over the 86.2 initialization)
NAVSIM v2	EPDMS	88.2
HUGSIM	Average HD-Score	38.8

Matched Futures for Alternative Actions

Simulator-rendered futures for going straight (A), changing lanes left (B), and changing lanes right (C).

Video-Trajectory Alignment

Video-trajectory alignment: World Engine renderings compared with SimVTA and Epona predictions.

Acknowledgements

We thank the authors and open-source contributors of the following projects for providing the models, simulation tools, and evaluation benchmarks that support this work:

• Epona | World Engine | NAVSIM | HUGSIM

We also thank the broader open-source community for its contributions to driving world models, autonomous driving, and 3D Gaussian Splatting.

License and Citation

All content in this repository is licensed under the Apache-2.0 license.

If our paper or code helps your research, please consider citing our work and giving this repository a star.