Tequila7437/EcoVGGT
Tequila7437/EcoVGGT has 12 GitHub stars and 0 forks. GitHot measured +3 stars in 24 hours. Last checked 2026-11-15 22:08 UTC. Last push 2026-11-13.
Measured star growth
- in 24h
- +3
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
EcoVGGT: pinned-key inference for efficient and streamable feed-forward 3D reconstruction from long videos
Machine LearningREADME.md
OriginalEco3R: Pinned-Key Inference for Efficient and Streamable
Feed-Forward 3D Reconstruction from Long Videos
Hongsen Cao · Shanxin Yuan · Mona Jaber · Ahmed Sayed
Queen Mary University of London

Eco3R (π³) reconstructs a 1,000-frame ScanNet video while the frames arrive.
News
- 2026-11: Code, Alignment Refocusing weights, and paper released.
Contents
Overview · Results · Installation · Quick start · Backbones · Online Stream · Alignment Score · Evaluation · Training · Model weights · Citation · License
Overview
VGGT and similar feed-forward models reconstruct a video in a single pass, but their cross-frame self-attention grows quadratically with the number of frames, and their accuracy drops on long videos. In Eco3R, every frame attends to the same 31 frames, spread evenly over the video. We call these frames the Pinned Key Set. A pin pass caches their tokens in a Scene Cache, and the remaining frames read this cache in independent batches. The result equals joint inference with these keys up to floating-point rounding, while the cost per frame and the peak memory stay constant as the video grows. The Online Stream applies the same idea while a video is still being recorded.
Global blocks 10 to 15 of VGGT align the frames. The Alignment Score reads their attention to flag likely failures without training or ground truth. Alignment Refocusing fine-tunes only these six blocks (76M parameters, 3.6 GPU hours) and reduces the failures. Eco3R runs on VGGT, π³, VGGT-Ω, and Depth Anything 3 (DA3), and each version carries the name of its backbone.

Results
On 1,000-frame ScanNet videos, Eco3R (π³) keeps the success rate of π³ and runs 13.8 times as fast, where success means an absolute trajectory error (ATE) below 0.2 m. On 500 frames, Eco3R (VGGT) with Alignment Refocusing raises the success rate of VGGT from 70% to 82% and runs 8.5 times as fast. Eco3R reconstructs 4,000 frames on one GPU, and Eco3R (π³) streams 1,000 frames at 18.6 frames per second.
| Method | 500 frames | 1,000 frames | ||||||
|---|---|---|---|---|---|---|---|---|
| Success (%) | ATE (m) | Time (s) | Memory (GB) | Success (%) | ATE (m) | Time (s) | Memory (GB) | |
| VGGT | 70 | 0.111 | 196 | 23.1 | 70 | 0.130 | 772 | 44.2 |
| Eco3R (VGGT) | 82 | 0.090 | 23.1 | 10.0 | 82 | 0.093 | 46.5 | 10.0 |
| Eco3R (VGGT), online | 78 | 0.102 | 45.8 | 11.4 | 78 | 0.099 | 64.6 | 11.5 |
| π³ | 94 | 0.054 | 138 | 24.4 | 94 | 0.058 | 557 | 43.4 |
| Eco3R (π³) | 94 | 0.052 | 19.9 | 9.9 | 94 | 0.060 | 40.4 | 9.9 |
| Eco3R (π³), online | 90 | 0.061 | 37.0 | 11.0 | 92 | 0.061 | 53.8 | 11.0 |
These rows come from Table 1 of the paper, which also covers 16 other methods. Each length has 50 ScanNet videos, and every method ran on one A100 GPU with 80 GB. Time is the median per video. Memory is the peak of inference over the videos, without the input frames.

A 4,401-frame ScanNet video seen from above, with the ceiling removed. InfiniteVGGT misplaces the walls, while Eco3R (π³) keeps every wall in place at 1,000 and 4,000 frames.

Success rate against speed at 1,000 frames, and time per frame and peak memory against video length.
Installation
git clone https://github.com/Tequila7437/Eco3R.git
cd Eco3R
conda create -n eco3r python=3.10 -y
conda activate eco3r
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -e .
bash scripts/setup_third_party.sh
The setup script clones FastVGGT and π³ into third_party/ at the commits we used. We use the VGGT implementation of FastVGGT, as in the paper. A few parts need extra steps:
- DA3:
bash scripts/setup_third_party.sh da3andpip install -e third_party/Depth-Anything-3 - VGGT-Ω:
bash scripts/setup_third_party.sh vggt-omega, and accept the license on its Hugging Face page before the first download - Training:
bash scripts/setup_third_party.sh vggtfor the VGGT training losses - Evaluation:
pip install -e ".[eval]"
We tested the code with Python 3.10 and PyTorch 2.11 on CUDA 12.8. The VGGT backbones run in bfloat16, which needs an NVIDIA GPU of the Ampere generation or newer.
Quick start
import eco3r
from eco3r.io import load_images
images = load_images("path/to/frames") # [S, 3, H, W] in [0, 1], 518 pixels wide
model = eco3r.load_backbone("pi3") # or "vggt", "vggt-frozen", "da3", "vggt-omega"
rec = eco3r.reconstruct(model, images.cuda(), "pi3")
rec["w2c"] # [S, 3, 4] world-to-camera matrices (OpenCV), world = camera of frame 0
rec["intrinsics"] # [S, 3, 3]
rec["depth"] # [S, H, W], confidence in rec["conf"]
rec["world_points"] # [S, H, W, 3]
load_images takes a folder of images, sorted by the numbers in their names, or a video file. For VGGT-Ω, call it with backbone="vggt-omega" to get the 416-pixel input of that model. Weights download on first use. To use local files, pass weights= and, for "vggt", refocusing_weights= to load_backbone.
The demo writes the cameras (cameras.json), the depth maps (depth.npz), and a colored point cloud (points.ply):
python demo.py --input path/to/frames --backbone pi3 --out outputs/scene
python demo.py --input video.mp4 --stride 5 --backbone vggt --score --out outputs/video
python demo.py --input path/to/frames --backbone pi3 --online --key_interval 32 --out outputs/stream
Backbones
| Name | Network | Offline | Online Stream | Alignment Score |
|---|---|---|---|---|
vggt |
VGGT-1B with Alignment Refocusing | yes | yes | yes |
vggt-frozen |
VGGT-1B with its released weights | yes | yes | yes |
pi3 |
π³ | yes | yes | no |
da3 |
DA3-LARGE | yes | no | no |
vggt-omega |
VGGT-Ω, 416-pixel checkpoint for benchmarking | yes | yes | no |
The VGGT backbones use bfloat16 weights. π³ and DA3 keep float32 weights and run under bfloat16 autocast, and VGGT-Ω follows its official forward pass. By default, 31 frames are pinned and the others are attached in batches of 64 (num_keys and chunk in eco3r.reconstruct). The GPU memory of a pass depends on these two numbers only, not on the length of the video.
Online Stream
The Online Stream takes frames one at a time. Every key_interval frames, the newest frame becomes a candidate key, and each pass pins up to max_keys candidates spread over the past. A frame is output lag passes after its interval, so it also sees up to lag + 1 keys from later in the video. No frame waits longer than (lag + 1) * key_interval frames.
stream = eco3r.Pi3OnlineStream(model, key_interval=32, max_keys=32)
for image in frames: # [3, H, W] in [0, 1] on the GPU
emitted = stream.push(image) # frames finished by this call
stream.finish() # flush the rest
out = stream.results()
eco3r.VGGTOnlineStream(model, key_interval, lag=2, max_keys=32) has the same interface for the VGGT backbones. For VGGT-Ω, eco3r.vggt_omega_engine.online_reconstruct runs the stream over a recorded clip. demo.py --online shows how to turn the outputs into cameras and points. In the paper, key_interval is 16 up to 500 frames and 32 beyond, max_keys is 32, and lag is 2 for VGGT and 0 for π³.
Alignment Score
model = eco3r.load_backbone("vggt")
rec = eco3r.reconstruct(model, images.cuda(), "vggt", score=True)
if rec["alignment_score"] < eco3r.THRESHOLD_PINNED_KEYS:
print("the reconstruction has likely failed")
The score is the 90th percentile of the divergence between each query's attention and the uniform distribution, over 2,048 random queries and all heads of global block 13 in the pin pass. It needs no training or ground truth, and it is ready before the first attach batch. Low values flag likely failures. For 500-frame ScanNet videos we use 9.09 with the Pinned Key Set and 7.83 with full attention, and the best threshold shifts on longer videos. The score works for the VGGT backbones and does not detect the failures of π³.
Evaluation
Download the ScanNet .sens files after accepting the terms of use, and export the 50 evaluation scenes:
python eval/prepare_scannet.py --sens_root /data/scannet/scans --scenes eval/scannet_50.txt \
--out /data/scannet/process_scannet
Each of these commands evaluates one row of Table 1:
python eval/eval_scannet.py --data_dir /data/scannet/process_scannet --sens_root /data/scannet/scans \
--backbone vggt --num_frames 500 --out results/vggt_500
python eval/eval_scannet.py --data_dir /data/scannet/process_scannet --sens_root /data/scannet/scans \
--backbone pi3 --num_frames 1000 --online --out results/pi3_online_1000
Frames are sampled as in FastVGGT: the first frame, then every k-th frame with the largest k that fills the budget. The script saves the metrics of each scene and a summary with the success rate and the medians of the ATE, the relative rotation and translation errors (RPE-r and RPE-t), point accuracy and completeness, the absolute relative depth error (AbsRel), time, and peak memory. Point metrics use the predicted world points for π³ and DA3 and the unprojected depth maps for VGGT and VGGT-Ω, as in the paper. Without --sens_root, the script computes the trajectory metrics only, which is all the paper reports at 2,000 and 4,000 frames besides time and memory. Our timings come from A100 GPUs with 80 GB. Other GPUs can give slightly different metrics, because their bfloat16 kernels differ.
Training Alignment Refocusing
bash scripts/setup_third_party.sh vggt
python eval/prepare_scannet.py --sens_root /data/scannet/scans --scenes training/scannet_train_195.txt \
--out /data/scannet/process_scannet_train --train
python training/train_refocusing.py --data_dir /data/scannet/process_scannet_train \
--scenes training/scannet_train_195.txt --weights /path/to/VGGT-1B/model.pt --out runs/refocusing
The 195 training scans are the first scans of ScanNet rooms that are not among the evaluation scenes. Each step takes a whole scan or a random contiguous half of it, pins 24 evenly spaced frames, and adds 24 random frames that attend only to the pinned ones. The loss is the camera loss of VGGT on all frames plus its depth loss on six of them. Our weights come from 2,000 AdamW steps at a learning rate of 1e-5, which took 3.6 hours on one A100 GPU and 37 GB of memory. To load your own checkpoint, call eco3r.load_backbone("vggt", refocusing_weights="runs/refocusing/alignment_refocusing_step2000.safetensors").
Model weights
The Alignment Refocusing weights (alignment_refocusing_vggt1b.safetensors, 302 MB) replace global blocks 10 to 15 of VGGT-1B and hold 75.6M parameters. They are on Hugging Face, and eco3r.load_backbone("vggt") downloads them together with VGGT-1B. The other backbones use the weights released by their authors.
Citation
@article{cao2026eco3r,
title = {{Eco3R}: Pinned-Key Inference for Efficient and Streamable Feed-Forward {3D} Reconstruction from Long Videos},
author = {Cao, Hongsen and Yuan, Shanxin and Jaber, Mona and Sayed, Ahmed},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
year = {2026}
}
Acknowledgements
Eco3R builds on VGGT, FastVGGT, π³, VGGT-Ω, and Depth Anything 3, and its evaluation uses ScanNet, evo, and Open3D. We thank their authors for making them public.
License
The code derives from VGGT and is released under the VGGT License. Files adapted from π³, Depth Anything 3, and VGGT-Ω keep their own licenses, listed in NOTICE.md, and the license of VGGT-Ω restricts its engine to non-commercial research. The Alignment Refocusing weights are released under CC BY-NC 4.0, the license of VGGT-1B, from which they are fine-tuned.