Stream video to reconstruct real-time 3D spaces
With only video captured by a camera, you can convert the scene into a 3D point cloud in real time and view it directly in a browser. This is a streaming 3D reconstruction model that calculates 3D structure immediately as video frames arrive. It obtains results with a single forward pass without complex optimization, enabling stable inference even for long videos.
ECCV 2026 Oral Presentation and Recent Updates
This project was selected as an oral presentation and a best paper candidate at ECCV 2026. Additionally, evaluation scripts for the KITTI and Oxford Spires datasets were released on May 25.
How the Geometric Context Transformer Works
The model uses a single streaming framework that combines coordinate-based anchors, pose reference windows, and trajectory memory. This integrates coordinate alignment, dense geometric cues, and long-range drift correction within a single structure. By applying a feedforward architecture and paged KV cache attention, it maintains a stable inference speed of approximately 20 FPS for long sequences exceeding 10,000 frames at a resolution of 518×378. It demonstrates superior reconstruction performance across various benchmarks compared to existing streaming or iterative optimization methods.
Interactive Demo and Offline Rendering Pipeline
Short videos can be explored in real time as 3D point clouds in a browser-based viser viewer by running demo.py. For long videos exceeding 3,000 frames, use demo_render/batch_demo.py to generate headless point cloud flythrough MP4 files. The offline pipeline allows designing camera paths (tracking, bird's eye, fixed, etc.) via YAML configuration files, enabling the staging of 3D reconstruction results like video editing. Various examples are provided, including indoor walking, outdoor driving, and videos generated by a proprietary world model.
Installation and Execution Environment
Python 3.10, PyTorch 2.8.0, and a CUDA 12.8 environment are recommended. Install the package with pip install -e ., and it is advisable to install FlashInfer for efficient streaming inference. If FlashInfer is not available, it falls back to PyTorch native attention (SDPA). To use offline rendering, you must additionally build Kaolin, ffmpeg, and CUDA extension modules. Model weights can be downloaded from Hugging Face or ModelScope.
Considerations for Long Sequence Processing
Since the model was trained on 320 views, storing more than 320 views in the KV cache degrades performance. To address this, use --keyframe_interval to store only every Nth frame as a keyframe, or use --mode windowed for sequences of 3,000 frames or more. State reset is not performed by default, so moving beyond the maximum distance seen during training may cause pose collapse. In such cases, switch to windowed mode or adjust the keyframe interval. It is distributed under the Apache License 2.0, and the ECCV 2026 paper should be cited when used for research purposes.
