Hanxun Yu1,2*,
Xuan Qu1,2*,
Lei Ke2,
Boqiang Zhang2,
Yuxin Wang2,3,
Jianke Zhu1,4,
Dong Yu2
1Zhejiang University,
2Tencent Hunyuan,
3HKUST,
4Shenzhen Loop Area Institute
project_video.mp4
Stream3D-VLM is an online 3D vision-language model that supports real-time spatial understanding and interaction directly from streaming video. By incrementally integrating geometry priors and employing geometry-adaptive voxel compression, our approach enables efficient and continuous 3D scene comprehension without requiring offline processing or complete scene observations.
- [2026-06-18] π₯ Stream3D-VLM is accepted by ECCV 2026!
- [2026-06-08] π₯ We release Stream3D-1M Dataset and Stream3D-Bench in Hugging Face π€.
- [2026-06-08] π₯ We release the checkpoint of Stream3D-VLM-4B in Hugging Face π€.
- [2026-06-08] π₯ We release the training and inference code.
- [2026-06-05] π₯ We release the paper of Stream3D-VLM.
git clone https://github.com/hanxunyu/Stream3D-VLM.git
cd Stream3D-VLM
export PYTHONPATH=$(pwd)/src:$PYTHONPATH
conda create -n stream3d-llm python=3.10 -y
conda activate stream3d-llm
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1 --no-build-isolation
Illustration of our data generation pipeline. Guided by a comprehensive task taxonomy spanning five cognitive competencies and three temporal interaction modes, the pipeline leverages detailed metadata from RGB-D video streams and a hybrid generation strategy to construct a large-scale spatio-temporal 3D QA dataset and the Stream3D-Bench for evaluating online 3D spatial understanding.
- Due to licensing restrictions, we are unable to directly release the source media. Instead, we provide annotations for the Stream3D-1M Dataset and Stream3D-Bench. Please refer to dataset_description.md for detailed dataset information.
We provide the pretrained model Stream3D-VLM-4B in Hugging Face π€.
Run our example inference script:
bash scripts/eval/eval_examples.shTo evaluate on Stream3D-Bench, set --data_path to the benchmark annotation file and --image_root to the directory containing the source media, then run:
bash scripts/eval/eval_bench.shAfter inference, set input_file in the corresponding metric script to the generated result file and compute the metrics:
# NA and MCA metrics
python scripts/comput_metrics/compute_score-NA+MCA.py
# OEA metric
# Set `api_key` and `base_url` before running.
python scripts/comput_metrics/compute_score-OEA.py
# Answer-Timing Accuracy (ATA)
python scripts/comput_metrics/compute_answer-timing-accuracy.py
# Response latency (TTFT and end-to-end latency)
python scripts/comput_metrics/compute_response-latency.pyConfigure the datasets in src/qwen_vl/data/__init__.py, set DATASETS in the training script, and run:
bash scripts/train/train_stream3d-vlm.shWe use StreamVGGT to generate point clouds that visualize the streaming 3D perception process. Two output formats are supported:
- GLB β incremental step-by-step point clouds viewable in any 3D viewer
- RRD β Rerun recording with camera poses and point clouds
# For GLB output:
pip install roma trimesh scipy matplotlib
# For RRD output:
pip install roma rerun-sdk
# Checkpoint will be auto-downloaded from HuggingFace on first run,
# or specify a local path via --ckpt.GLB (incremental point clouds):
bash visualization/run_incremental_reconstruction.shRRD (Rerun recording):
bash visualization/run_incremental_reconstruction_rrd.shEdit the scripts to configure input/output paths and parameters (e.g., INPUT_DIR, CKPT_PATH, STEP).
- GLB:
incremental_steps/step_XXXX.glb(each incremental step) andscene_final_conf{thres}.glb(final full reconstruction) - RRD:
scene_final_conf{thres}.rrd(final reconstruction with camera poses and point clouds), viewable via:rerun output/scene_final_conf50.0.rrd
backward_converted.mp4
realtime_converted.mp4
forward_converted.mp4
We are grateful for the open-source contributions of other projects:
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
@article{yu2026stream3d,
title={Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors},
author={Hanxun Yu and Xuan Qu and Lei Ke and Boqiang Zhang and Yuxin Wang and Jianke Zhu and Dong Yu},
journal={arXiv preprint arXiv:2606.06891},
year={2026}
}
