Skip to content
Projects

neoteai/N0-VTLA has 204 GitHub stars and 11 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-18 10:27 UTC. Last push 2026-08-17.

How these counts are measured

What you can do

  • Post-train the pretrained checkpoint on your own visuo-tactile demonstrations with train.sh. See POST_TRAINING.md.
  • Convert your episodes to the canonical 32-dim layout and compute normalization statistics (scripts/convert_canonical_data.py, scripts/compute_canonical_norm.py).
  • Serve a policy over websocket for a real robot, or over ZMQ for the simulator (scripts/serve_policy.py, scripts/serve_zmq.py). See DEPLOY.md.
  • Check that the tactile pathway is doing something, and that the model stays byte-equivalent to the base policy with tactile off (scripts/probe_z_tactile_dependence.py, scripts/gate_c_check.py).

Measured star growth

in 24h
0
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

N0-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

Developer Tools

README.md

Original

𝒩₀-VTLA

Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

English  ·  中文

Project Page Paper Weights License: CC BY-SA 4.0 Contact: hengzzzhou@gmail.com Python 3.11

🔥 𝒩₀-VTLA Has Been Released! 🔥

Pretrained checkpoints, the post-training toolkit, and inference servers are now available.

N0-VTLA: large-scale visuo-tactile pretraining with latent tactile tokens

$\mathcal{N}_0$-VTLA is a vision–tactile–language–action policy that gives a pretrained vision–language–action (VLA) model a sense of touch. Rather than feeding tactile images in as extra camera streams, it predicts latent tactile tokens, a compact code for the contact change expected over the next action chunk, and conditions a flow-matching action expert on them directly. Force, slip, and the difference between alignment and jamming are visually subtle but directly observable through touch; the latent tactile pathway makes that signal available to the policy.

This repository releases the model, the post-training toolkit, and the inference servers, so you can load our pretrained weights and adapt $\mathcal{N}_0$-VTLA to your own robot and tasks. It does not ship the large-scale pretraining pipeline.

Capabilities

  • Post-train the pretrained checkpoint on your own visuo-tactile demonstrations with train.sh. See POST_TRAINING.md.
  • Convert your episodes to the canonical 32-dim layout and compute normalization statistics (scripts/convert_canonical_data.py, scripts/compute_canonical_norm.py).
  • Serve a policy over websocket for a real robot, or over ZMQ for the simulator (scripts/serve_policy.py, scripts/serve_zmq.py). See DEPLOY.md.
  • Check that the tactile pathway is doing something, and that the model stays byte-equivalent to the base policy with tactile off (scripts/probe_z_tactile_dependence.py, scripts/gate_c_check.py).

Model summary

Component Choice
Backbone PaliGemma (gemma_2b) prefix + Gemma (300m) action expert
Objective Flow matching over a 50-step action chunk
Tactile encoder Frozen DINOv2 (facebook/dinov2-base) over baseline-difference images
Tactile pathway Cross-attention predictor → 5 latent tokens, injected into the action expert
Action space Canonical 32-dim container (dual-arm EEF position + rot6d + gripper)
Precision bf16 parameters, eager attention to match the pretraining path

Latent tactile model overview
Each tactile frame is differenced against a zero-contact baseline, encoded by a frozen DINOv2, and distilled into a few latent tokens summarizing the contact change expected over the next action chunk.

The tactile path is off by default and byte-equivalent to the base policy when disabled, so an $\mathcal{N}_0$-VTLA checkpoint loads and runs like the standard base policy until the tactile branch is switched on. scripts/gate_c_check.py asserts that equivalence on state-dict keys, forward loss, and sampled actions.

Repository structure

N0-VTLA/
├── n0vtla/                 # the model package
│   ├── models_pytorch/     # N0VTLAPolicy, tactile encoder/predictor, transformers_replace
│   ├── policies/           # input/output transforms per embodiment, canonical schema
│   ├── training/           # configs, data loader, optimizer
│   └── transforms.py       # normalization, delta actions, tokenization
├── n0vtla_client/          # robot-side client library
├── scripts/                # train, serve, convert, verify
├── docs/                   # installation, post-training, deployment
└── train.sh                # post-training launcher

Installation

Python 3.11 with a CUDA-capable GPU. The transformers_replace patch and the DINOv2 cache are both required; INSTALL.md has the full procedure.

conda create -n vtla python=3.11 -y && conda activate vtla
pip install -r requirements.txt && pip install -e . --no-deps
cp -r n0vtla/models_pytorch/transformers_replace/* \
  "$CONDA_PREFIX/lib/python3.11/site-packages/transformers/"

📦 Model Download

Checkpoint What it is Config
🤗 n0-vtla-base pretrained base, the starting point for post-training vtla_tactile_posttrain
🤗 n0_VTLA_insert_hole UniVTAC single-arm policy, joint actions sim_single_arm_tactile
🤗 n0_VTLA_dual_bowl_place_stack NeoSim dual-arm policy, joint actions sim_dual_arm_tactile
hf download NeoteAI/n0-vtla-base --local-dir checkpoints/n0-vtla-base
export VTLA_PRETRAINED_CHECKPOINT="$PWD/checkpoints/n0-vtla-base"

VTLA_PRETRAINED_CHECKPOINT must name the directory that directly contains model.safetensors.

The base checkpoint is a pretrained base for post-training, not a deployable policy. It carries the tactile encoder, latent tactile predictor, and projection parameters that a task policy needs, but it has not been fine-tuned on any task.

Quick start: post-train on your own data

export VTLA_DATASET_PATH=/path/to/datasets/canonical_tactile_task
export VTLA_ASSET_ID=canonical_tactile_task
export EXP_NAME=my_experiment

CHECK_ONLY=1 bash train.sh     # preflight: GPUs, checkpoint, dataset, norm stats, DINOv2 cache
bash train.sh --overwrite

POST_TRAINING.md covers the four stages (convert episodes, compute normalization statistics, configure, train), along with the canonical 32-dim layout and a troubleshooting section.

Serving

# real robot, websocket
python scripts/serve_policy.py \
  --policy.config=vtla_tactile_posttrain \
  --policy.dir=checkpoints/vtla_tactile_posttrain/<experiment>/<step>

# UniVTAC simulator, ZMQ
VTLA_ASSET_ID=n0_insert_hole_norm python scripts/serve_zmq.py \
  --config sim_single_arm_tactile \
  --ckpt checkpoints/n0_VTLA_insert_hole \
  --addr "tcp://*:5557" --default-prompt "insert hole"

Execute the full action chunk before requesting the next prediction. A shorter execution stride drops the tail of every chunk, and the gripper-close commands live in that tail. See DEPLOY.md.

Highlights

  • Predictive touch, not pixels. Tactile evidence enters as a small set of predicted latent tokens for upcoming contact change, rather than as raw images decoded through the vision tower.
  • Baseline-difference input. Each tactile frame is subtracted from the episode's zero-contact baseline, so the encoder sees contact change rather than the raw gel image.
  • Zero-initialized gate. The latent tokens enter the action expert through a gate that starts at zero, so training begins from the exact base-policy behavior and opens the tactile channel gradually.
  • One action space for many robots. A fixed 32-dim container lets single-arm, dual-arm, and handheld-gripper embodiments share a single action objective.

Large-scale pretraining on NeoData, the three-stage training recipe, the deployment-time (RL) improvement pipeline, and the experimental results are described in the paper.

Contact

For questions about $\mathcal{N}_0$-VTLA, you can contact [email protected].

Citation

@misc{n0vtla2026,
      title={$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens}, 
      author={NeoteAI Team and Fudan TEAI Team},
      year={2026},
      eprint={2607.23782},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.23782}, 
}

Acknowledgments

$\mathcal{N}_0$-VTLA builds on the OpenPI framework, DINOv2 for tactile encoding, and LeRobot for data handling.

License

Original material in this repository is released under CC BY-SA 4.0. Third-party components retain their original licenses; the Apache-2.0-licensed components are identified in NOTICE, with the full licence text in LICENSES/Apache-2.0.txt.

Model weights are not covered by that licence. The released $\mathcal{N}_0$-VTLA checkpoints derive from Google's PaliGemma/Gemma parameters, so they are made available under the Gemma Terms of Use and the Gemma Prohibited Use Policy. This is inherited from the base model, not a restriction we add on top.