Skip to content

About

[ECCV 2026πŸ”₯] Official code repository for "Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors"

Topics

Resources

Stars

74 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

108 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Stream3D-VLM Logo Β Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

[ECCV 2026]

Hanxun Yu1,2*, Xuan Qu1,2*, Lei Ke2, Boqiang Zhang2, Yuxin Wang2,3, Jianke Zhu1,4, Dong Yu2
1Zhejiang University, 2Tencent Hunyuan, 3HKUST, 4Shenzhen Loop Area Institute

project_video.mp4

πŸ” Overview

model

Stream3D-VLM is an online 3D vision-language model that supports real-time spatial understanding and interaction directly from streaming video. By incrementally integrating geometry priors and employing geometry-adaptive voxel compression, our approach enables efficient and continuous 3D scene comprehension without requiring offline processing or complete scene observations.

πŸ“° News

  • [2026-06-18] πŸ”₯ Stream3D-VLM is accepted by ECCV 2026!
  • [2026-06-08] πŸ”₯ We release Stream3D-1M Dataset and Stream3D-Bench in Hugging Face πŸ€—.
  • [2026-06-08] πŸ”₯ We release the checkpoint of Stream3D-VLM-4B in Hugging Face πŸ€—.
  • [2026-06-08] πŸ”₯ We release the training and inference code.
  • [2026-06-05] πŸ”₯ We release the paper of Stream3D-VLM.

πŸ› οΈ Installation

git clone https://github.com/hanxunyu/Stream3D-VLM.git
cd Stream3D-VLM
export PYTHONPATH=$(pwd)/src:$PYTHONPATH

conda create -n stream3d-llm python=3.10 -y
conda activate stream3d-llm
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1 --no-build-isolation

πŸ“Š Datasets and Benchmark

data_generation

Illustration of our data generation pipeline. Guided by a comprehensive task taxonomy spanning five cognitive competencies and three temporal interaction modes, the pipeline leverages detailed metadata from RGB-D video streams and a hybrid generation strategy to construct a large-scale spatio-temporal 3D QA dataset and the Stream3D-Bench for evaluating online 3D spatial understanding.

πŸ“¦οΈ Pretrained Models

We provide the pretrained model Stream3D-VLM-4B in Hugging Face πŸ€—.

πŸ€– Inference Examples

Run our example inference script:

bash scripts/eval/eval_examples.sh

To evaluate on Stream3D-Bench, set --data_path to the benchmark annotation file and --image_root to the directory containing the source media, then run:

bash scripts/eval/eval_bench.sh

After inference, set input_file in the corresponding metric script to the generated result file and compute the metrics:

# NA and MCA metrics
python scripts/comput_metrics/compute_score-NA+MCA.py

# OEA metric
# Set `api_key` and `base_url` before running.
python scripts/comput_metrics/compute_score-OEA.py

# Answer-Timing Accuracy (ATA)
python scripts/comput_metrics/compute_answer-timing-accuracy.py

# Response latency (TTFT and end-to-end latency)
python scripts/comput_metrics/compute_response-latency.py

πŸš€ Training

Configure the datasets in src/qwen_vl/data/__init__.py, set DATASETS in the training script, and run:

bash scripts/train/train_stream3d-vlm.sh

πŸŽ₯ Visualization

🌐 Incremental 3D Reconstruction

We use StreamVGGT to generate point clouds that visualize the streaming 3D perception process. Two output formats are supported:

  • GLB β€” incremental step-by-step point clouds viewable in any 3D viewer
  • RRD β€” Rerun recording with camera poses and point clouds

Setup

# For GLB output:
pip install roma trimesh scipy matplotlib

# For RRD output:
pip install roma rerun-sdk

# Checkpoint will be auto-downloaded from HuggingFace on first run,
# or specify a local path via --ckpt.

Usage

GLB (incremental point clouds):

bash visualization/run_incremental_reconstruction.sh

RRD (Rerun recording):

bash visualization/run_incremental_reconstruction_rrd.sh

Edit the scripts to configure input/output paths and parameters (e.g., INPUT_DIR, CKPT_PATH, STEP).

Output

  • GLB: incremental_steps/step_XXXX.glb (each incremental step) and scene_final_conf{thres}.glb (final full reconstruction)
  • RRD: scene_final_conf{thres}.rrd (final reconstruction with camera poses and point clouds), viewable via:
    rerun output/scene_final_conf50.0.rrd

πŸ•˜ Demo 1-Backward Tracing (Memory)

backward_converted.mp4

πŸ‘οΈ Demo 2-Realtime Perception (Observation)

realtime_converted.mp4

⏩ Demo 3-Forward Response (Monitoring)

forward_converted.mp4

πŸ‘ Acknowledgements

We are grateful for the open-source contributions of other projects:

πŸ“‘ License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

πŸ–ŠοΈ Citation

@article{yu2026stream3d,
    title={Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors},
    author={Hanxun Yu and Xuan Qu and Lei Ke and Boqiang Zhang and Yuxin Wang and Jianke Zhu and Dong Yu},
    journal={arXiv preprint arXiv:2606.06891},
    year={2026}
}

About

[ECCV 2026πŸ”₯] Official code repository for "Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors"

Topics

Resources

Stars

74 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages