Skip to content
Projekte

irajisamurai/2D-Motion-Interface has 26 GitHub stars and 1 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 15:17 UTC. Last push 2026-08-21.

So werden diese Zahlen gemessen

What you can do

  • 🔌 Plug-and-play, at very low training cost. The interface is an external module: no modification and no fine-tuning of the pretrained MoLM. E_2D has 9.6M parameters (~22 h) and the real-video adapter A_real only 0.3M (~1 h), both on a single A100 40GB.
  • 📈 Comparable to 3D motion inputs. On HumanML3D motion captioning, Avg Drop vs 3D Input is −0.2% (TM2T), +0.4% (MotionGPT), and −2.8% (MG-MotionLLM).
  • 🥊 Better than training MoLMs from scratch on 2D. 2D Scratch drops −25.8% (TM2T) and −12.8% (MG-MotionLLM) versus 3D Input — our interface substantially reduces that gap at a fraction of the compute.
  • 🔬 2D and 3D motions align in both latent and token space. Discrete token agreement with 3D-derived tokens reaches 44.8% / 59.6% / 67.1% (Top-1/2/3), against a 0.20% random baseline for a codebook of size 512.

Gemessener Sternzuwachs

in 24h
0
in 7 days
Aktualisierung verzögert
in 15 days
Aktualisierung verzögert
in 1 month
Aktualisierung verzögert

Nettoänderungen in überlappenden Zeiträumen, keine Tagessummen. Fehlende Messwerte bleiben leer. Sterne zeigen Interesse, nicht Qualität.

Über uns

Plug-and-play 2D motion interface for real-world Motion Language Models

Maschinelles Lernen

README.md

Original

🕺 A Plug-and-Play 2D Motion Interface
for Real-World Motion Language Models

Kaname Yokoyama1 · Norimichi Ukita1
1 Toyota Technological Institute

🏆 Accepted to HCMIW @ ECCV 2026 — Oral Presentation

Venue Oral

arXiv Project Page Live Demo Dataset Checkpoints

Python PyTorch CUDA Stars


🎬 Demo

Demo: a monocular real video of a person sitting down cross-legged and standing back up, with the estimated ViTPose 2D keypoints overlaid, and the caption predicted by our 2D Motion Interface shown underneath.

A monocular real video with the estimated ViTPose 2D keypoints overlaid. The caption underneath is generated by our 2D Motion Interface — no 3D pose estimation anywhere in the pipeline.

✨ Highlights

  • 🔌 Plug-and-play, at very low training cost. The interface is an external module: no modification and no fine-tuning of the pretrained MoLM. E_2D has 9.6M parameters (~22 h) and the real-video adapter A_real only 0.3M (~1 h), both on a single A100 40GB.
  • 📈 Comparable to 3D motion inputs. On HumanML3D motion captioning, Avg Drop vs 3D Input is −0.2% (TM2T), +0.4% (MotionGPT), and −2.8% (MG-MotionLLM).
  • 🥊 Better than training MoLMs from scratch on 2D. 2D Scratch drops −25.8% (TM2T) and −12.8% (MG-MotionLLM) versus 3D Input — our interface substantially reduces that gap at a fraction of the compute.
  • 🔬 2D and 3D motions align in both latent and token space. Discrete token agreement with 3D-derived tokens reaches 44.8% / 59.6% / 67.1% (Top-1/2/3), against a 0.20% random baseline for a codebook of size 512.
  • 🎥 On real monocular video, 2D beats estimated 3D. 2D Input w/ A_real outperforms all 3D Input settings on almost all metrics across all three MoLMs — while being far cheaper: ViTPose (Base) needs 17.2 GFLOPs/frame versus 250.2 for WHAM.
  • 🗂 New monocular real-world video motion dataset. 132 videos from 10 participants (14,878 frames / 743.9 s), paired with estimated 2D (ViTPose) and 3D (TRACE, WHAM) motions and HumanML3D captions; 86.4% of video–caption pairs were human-verified as semantically consistent. Publicly released.
📑 Table of Contents

🔍 How It Works

Overview. (a) Training stage: a 3D motion goes through the MoLM's frozen tokenizer encoder E_3D, while a view-randomized 2D projection of the same motion goes through a new encoder E_2D; E_2D is trained with an L1 loss to reproduce the 3D latent. (b) Inference stage: 2D motion alone goes through E_2D into the original codebook, and the resulting motion tokens are concatenated with text tokens and decoded into a caption by the pretrained language model.

A new 2D motion encoder E_2D is trained to map 2D motion into the same latent space as the MoLM's own 3D encoder, with a simple L1 loss on the latents. The codebook, decoder, and language model stay frozen and are reused as-is, so 2D motion is tokenized by the original codebook and captioned by the pretrained language model — with no 3D pose estimation in the loop.

🎥 On real video: the adapter

Adapter overview. (a) Data collection: AMASS motions provide ground-truth 3D and are also rendered to video, run through ViTPose to obtain estimated 2D pose sequences with confidences, yielding pseudo-real training pairs. (b) Adapter training: the GT 3D motion goes through the frozen E_3D while the estimated 2D pose and confidences go through the trainable adapter A_real into the frozen E_2D, supervised by an L1 loss between the two latents.

E_2D is trained on clean 2D projections of mocap, while real video adds perspective distortion and pose-estimation noise. A small real-video adapter A_real (0.3M params) inserted before E_2D absorbs that gap. Since real videos have no ground-truth 3D, it is trained on pseudo-real pairs: AMASS motions rendered from random viewpoints, run through ViTPose, and paired with their own 3D.

📊 Headline Results

Both tables report Avg Drop, the average percentage difference from the reference 3D setting across all metrics (higher is better, – is the reference row itself).

1 · Motion captioning on HumanML3D — Avg Drop vs 3D Input. Replacing 3D inputs with our 2D interface costs almost nothing, while training the MoLM from scratch on 2D costs a great deal.

Base model 2D Scratch Ours (2D Motion Interface)
TM2T −25.8% −0.2%
MotionGPT not reported † +0.4%
MG-MotionLLM −12.8% −2.8%

† The MotionGPT 2D-Scratch result is omitted because the official implementation could not be stably retrained; the same reproducibility issue is open in the official repository.

2 · Motion captioning on monocular real videos — Avg Drop vs Ref 3D Input. A_real is our real-video adapter; A_3D is the same adapter applied to WHAM's estimated 3D motion for a fair comparison.

Base model 3D (TRACE) 3D (WHAM) 2D 3D (WHAM) w/ A_3D 2D w/ A_real
TM2T −37.7% −35.9% −18.5% −10.8% +4.3%
MotionGPT −40.3% −34.1% −31.4% −21.1% −18.5%
MG-MotionLLM −50.7% −35.6% −46.7% −34.9% −17.5%

2D Input w/ A_real outperforms all 3D Input settings on almost all metrics across all models — and does so far more cheaply: excluding human detection, ViTPose (Base) requires 17.2 GFLOPs/frame against 250.2 for WHAM.

[!NOTE] Ref 3D Input is the original HumanML3D motion sampled when collecting each video. It is not the strict ground-truth 3D motion of what the participant actually performed, so it serves as a reference point rather than a hard upper bound.

Full per-metric tables are in Reproducing Key Results.

📁 Repository Layout

Directory Base Model Tasks Role in this work
2DMotionGPT/ MotionGPT M2T Trains the 2D encoder and both adapters — the source of the shared weights
2DTM2T/ TM2T M2T Reuses the 2D encoder and adapters above
2DMG-MotionLLM/ MG-MotionLLM M2T, M2DT Trains its own 2D encoder and adapters independently

M2T = motion-to-text · M2DT = motion-to-detailed-text

📦 Installation

Each sub-project uses its own conda environment — the PyTorch versions are mutually incompatible, so all three are needed to reproduce everything.

Sub-project Env name Python PyTorch CUDA
2DMotionGPT/ mgpt_2d 3.10 2.9.0 11.8
2DTM2T/ tm2t 3.7 1.13.1 10.1
2DMG-MotionLLM/ mg-motionllm 3.9 2.8.0 11.8
cd <sub-project>                      # 2DMotionGPT | 2DTM2T | 2DMG-MotionLLM
conda env create -f environment.yml
conda activate <env-name>             # mgpt_2d | tm2t | mg-motionllm

One more environment is needed only to run ViTPose on your own videos (Caption your own video) — the released datasets already ship their 2D keypoints, so reproducing the paper does not need it:

Purpose Env name Python PyTorch CUDA Key packages
ViTPose 2D pose estimation openmmlab 3.8 1.9.1 11.1 mmpose 1.1.0 · mmdet 3.1.0 · mmcv 2.0.1
cd 2DMotionGPT
conda env create -f environment_openmmlab.yml

📂 Dataset Preparation

All data lives under 2DMotionGPT/dataset/:

2DMotionGPT/dataset/
├── HumanML3D/                     # HumanML3D
│   ├── new_joint_vecs/  texts/  Mean.npy  Std.npy
│   ├── train.txt  val.txt  test.txt  all.txt
│   └── finemotion_texts/          # FineMotion — 2DMG-MotionLLM only
├── real_world_dataset_ver2/       # our real-world video dataset (2D / ViTPose)
├── gt_upper_bound_wham/           # WHAM 3D motions of the same real videos
├── humanml3d_for_render/          # pseudo-real ViTPose 2D pairs — A_real training
└── humanml3d_for_render_wham/     # WHAM 3D pairs — A_3D training

2DTM2T/ and 2DMG-MotionLLM/ need no copy of their own — symlink them to the same data:

# from the repository root
mkdir -p 2DTM2T/dataset 2DMG-MotionLLM/dataset
ln -s ../../2DMotionGPT/dataset/HumanML3D 2DTM2T/dataset/HumanML3D
ln -s ../../2DMotionGPT/dataset/HumanML3D 2DMG-MotionLLM/dataset/HumanML3D

HumanML3D (all three sub-projects)

Follow the HumanML3D repository to download and prepare the dataset. Expected structure:

HumanML3D/
├── new_joint_vecs/
├── texts/
├── Mean.npy
├── Std.npy
├── train.txt
├── val.txt
├── test.txt
└── all.txt

FineMotion (2DMG-MotionLLM only)

Needed for the Motion-to-Detailed-Text task. Download BPMSD_auto.zip and BPMSD_human.zip from the FineMotion repository and place them under dataset/HumanML3D/finemotion_texts/:

dataset/HumanML3D/
├── finemotion_texts/
│   ├── BPMSD_auto.zip
│   ├── BPMSD_auto.json
│   ├── BPMSD_human.zip
│   └── BPMSD_human.json
└── ...

Monocular real-world video motion dataset (ours)

Motion clips were randomly sampled from the HumanML3D test set and shown to participants, who imitated them while being recorded with a single RGB camera (iPhone SE 2nd generation), varying the relative viewpoint between subject and camera. Each video is paired with estimated 2D motion (ViTPose), estimated 3D motion (TRACE and WHAM), and the original HumanML3D caption of the sampled clip. All performers participated with informed consent.

Videos 132, from 10 participants (male and female, 20s–50s)
Frames / duration 14,878 frames · 743.9 s
Captions original HumanML3D captions of the sampled clips
Human verification 3 annotators; 86.4% of video–caption pairs judged semantically consistent

📥 Download the real-world dataset (Google Drive)

Expected structure:

dataset/
├── HumanML3D/
├── gt_upper_bound_wham/
├── humanml3d_for_render/
├── humanml3d_for_render_wham/
└── real_world_dataset_ver2/

🧠 Pre-trained Checkpoints

Base models and dependencies

GloVe embeddings, evaluators, and base checkpoints come from the original repositories:

# MotionGPT
cd 2DMotionGPT
bash prepare/prepare_t5.sh
bash prepare/download_t2m_evaluators.sh
bash prepare/download_pretrained_models.sh

# MG-MotionLLM
cd 2DMG-MotionLLM
bash prepare/prepare_t5.sh
bash prepare/download_glove.sh
bash prepare/download_vqvae.sh

For MG-MotionLLM, also download the baseline checkpoints and place them in 2DMG-MotionLLM/:

Our checkpoints

📥 Download our 2D encoders and adapters (Google Drive)

⚡ Quick Start

Once Installation, the real-world dataset, and our checkpoints are in place, this single command reproduces the headline real-world result:

cd 2DMotionGPT
conda activate mgpt_2d
python evaluate_with_realworld.py \
    --cfg configs/config_h3d_stage1.yaml --nodebug \
    --estimated_motion_dir ./dataset/real_world_dataset_ver2/pred \
    --vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
    --adapter_ckpt ./checkpoints/adapter/MotionGPT_2DEncoder/adapter-residual-bs64-lr1e-3/best_adapter_epoch690_valloss1.2251.tar \
    --adapter_type residual \
    --mode both

--mode both evaluates with and without the adapter in one pass, so you get the comparison directly.

🎥 Caption your own video

demo/ runs the whole thing on an arbitrary monocular video — ViTPose 2D pose estimation, then captioning through A_real → E_2D → MotionGPT:

cd 2DMotionGPT
python demo/demo.py --video /path/to/your_video.mp4 --gpu_id 0

The driver only needs the standard library — it runs pose estimation in the openmmlab environment and captioning in mgpt_2d, since mmpose (torch 1.9) and MotionGPT (torch 2.9) cannot share one environment. Interpreters are found automatically; override with POSE_PYTHON / CAPTION_PYTHON if needed. Outputs land in demo/outputs/<video name>/: keypoints.json, overlay.mp4 (skeleton overlay), caption.json, pose_meta.json.

Only the weight bundle in space/bundle/ is required — no HumanML3D, no evaluators. The video is resampled to 20 fps automatically; use --start / --duration to pick a segment, since the model captions at most 196 frames (9.8 s). Best results come from a single person, full body in frame, roughly static camera.

Running the two stages separately
# stage 1 — openmmlab env
python demo/run_vitpose_single.py --video in.mp4 --out_json kp.json --out_video overlay.mp4
# stage 2 — mgpt_2d env
python demo/caption_from_json.py --json kp.json --mode both

📈 Reproducing Key Results

1. M2T Evaluation on HumanML3D (2D encoder vs 3D baseline)

2DMotionGPT — evaluate 2D encoder and 3D baseline side by side:

cd 2DMotionGPT
conda activate mgpt_2d
# 3D baseline
python evaluate_motionGPT.py --gpu_id 7 --eval_3d \
    --output results/eval_3d_bs1.json

# 2D encoder
python evaluate_motionGPT.py --gpu_id 7 \
    --vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
    --output results/eval_2d_bs1.json
📊 Paper Results — Table 1, MotionGPT (HumanML3D test set)
Method R-Prec Top-1↑ Top-2↑ Top-3↑ MM-Dist↓ BLEU-1↑ BLEU-4↑ ROUGE-L↑ CIDEr↑ BERTScore↑ Avg Drop vs 3D↑
3D Input 0.516 0.707 0.803 2.994 0.429 0.062 0.345 0.079 0.316 –
Ours 0.523
±0.002
0.717
±0.003
0.802
±0.003
2.985
±0.015
0.431
±0.000
0.063
±0.001
0.346
±0.001
0.077
±0.001
0.319
±0.001
+0.4%

Ours is the mean ± std over three runs. 2D Scratch is not reported for MotionGPT: the official implementation could not be stably retrained, an issue also open in the official repository.


2DMG-MotionLLM — evaluate 3D baseline and 2D encoder sequentially:

cd 2DMG-MotionLLM
# 3D baseline + 2D encoder (2D encoder is auto-detected from checkpoints/2d_vq_train/t2m/)
# M2T
python eval_m2t.py --model_name ./m2t-ft-from-GSPretrained-base
# M2DT
python eval_m2dt.py --model_name ./m2dt-ft-from-GSPretrained-base

# 2D scratch (full VQ-VAE trained from scratch on 2D data)
# M2T
python eval_m2t.py --model_name ./m2t-2drecon-100k/checkpoint-100000 \
    --vqvae_2drecon_ckpt ../2DMotionGPT/checkpoints/2d_vqvae_full/MotionGPT_2DEncoder/full_recon_2d-l1_smooth-bs64-lr0.0001/best_vqvae_epoch2990_valloss0.0354.tar
# M2DT
python eval_m2dt.py --model_name ./m2dt-2drecon/checkpoint-300000 \
    --vqvae_2drecon_ckpt ../2DMotionGPT/checkpoints/2d_vqvae_full/MotionGPT_2DEncoder/full_recon_2d-l1_smooth-bs64-lr0.0001/best_vqvae_epoch2990_valloss0.0354.tar
📊 Paper Results — Tables 1 & 2, MG-MotionLLM

M2T — motion captioning on the HumanML3D test set (Table 1):

Method R-Prec Top-1↑ Top-2↑ Top-3↑ MM-Dist↓ BLEU-1↑ BLEU-4↑ ROUGE-L↑ CIDEr↑ BERTScore↑ Avg Drop vs 3D↑
3D Input 0.583 0.787 0.873 2.571 0.507 0.091 0.402 0.095 0.386 –
2D Scratch 0.487 0.667 0.758 3.289 0.494 0.078 0.385 0.074 0.364 −12.8%
Ours 0.576 0.770 0.855 2.701 0.499 0.087 0.396 0.090 0.379 −2.8%

M2DT — motion-to-detailed-text on the FineMotion test set (Table 2):

Level Method BLEU-1↑ BLEU-4↑ BLEU-7↑ ROUGE-L↑ BERTScore↑ Avg Drop vs 3D↑
Sequence 3D Input 0.828 0.667 0.536 0.651 0.523 –
Sequence 2D Scratch 0.814 0.637 0.497 0.629 0.483 −4.9%
Sequence Ours 0.828 0.665 0.533 0.648 0.517 −0.5%
Snippet 3D Input 0.672 0.478 0.354 0.606 0.509 –
Snippet 2D Scratch 0.653 0.456 0.329 0.588 0.487 −4.4%
Snippet Ours 0.666 0.473 0.349 0.601 0.502 −1.1%

Ours rows are the mean over three runs (± ≤0.021 on every metric). The M2DT protocol follows MG-MotionLLM and reports BLEU-7 rather than CIDEr.


2. Real-world Evaluation

Requires the real-world video dataset (see Dataset Preparation).

2DMotionGPT:

cd 2DMotionGPT
# ViTPose 2D + 2D Adapter (main result)
python evaluate_with_realworld.py \
    --cfg configs/config_h3d_stage1.yaml --nodebug \
    --estimated_motion_dir ./dataset/real_world_dataset_ver2/pred \
    --vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
    --adapter_ckpt ./checkpoints/adapter/MotionGPT_2DEncoder/adapter-residual-bs64-lr1e-3/best_adapter_epoch690_valloss1.2251.tar \
    --adapter_type residual \
    --mode both

# WHAM 3D + 3D Adapter
python evaluate_with_realworld_3d_adapter.py \
    --cfg configs/config_h3d_stage1.yaml --nodebug \
    --estimated_motion_dir ./dataset/gt_upper_bound_wham \
    --adapter_ckpt ./checkpoints/adapter3d/MotionGPT_2DEncoder/adapter3d-residual-bs64-lr1e-3/best_adapter3d_epoch880_valloss1.2810.tar \
    --split test \
    --mode both
📊 Paper Results — Table 4, MotionGPT (monocular real-world video motion dataset)
Method BLEU-1↑ BLEU-4↑ ROUGE-L↑ CIDEr↑ BERTScore↑ Avg Drop vs Ref 3D↑
Ref 3D Input 0.461 0.069 0.405 0.149 0.352 –
3D Input (TRACE) 0.369 0.034 0.315 0.044 0.218 −40.3%
3D Input (WHAM) 0.380 0.035 0.339 0.056 0.264 −34.1%
2D Input 0.361 0.047 0.321 0.075 0.236 −31.4%
3D Input (WHAM) w/ A_3D 0.408 0.046 0.365 0.102 0.285 −21.1%
2D Input w/ A_real 0.435 0.045 0.367 0.105 0.306 −18.5%

Bold marks the best value among the estimated-motion rows (Ref 3D Input is a reference, not a competitor).


2DMG-MotionLLM:

cd 2DMG-MotionLLM
# ViTPose 2D + 2D Adapter (main result)
python evaluate_with_realworld.py \
    --estimated_motion_dir ../2DMotionGPT/dataset/real_world_dataset_ver2/pred \
    --adapter_ckpt ./checkpoints/adapter/MG-MotionLLM/best_adapter_epoch2520_valloss1.1380.pt \
    --adapter_type residual \
    --meta_dir ../2DMotionGPT/deps/t2m/t2m/VQVAEV3_CB1024_CMT_H1024_NRES3/meta \
    --data_root ./dataset/HumanML3D \
    --split test \
    --mode both \
    --gpu_id 0

# WHAM 3D + 3D Adapter
python evaluate_with_realworld_3d_adapter.py \
    --estimated_motion_dir ../2DMotionGPT/dataset/gt_upper_bound_wham \
    --adapter_ckpt ./checkpoints/adapter3d/MG-MotionLLM/best_adapter3d_epoch610_valloss1.1204.pt \
    --meta_dir ../2DMotionGPT/deps/t2m/t2m/VQVAEV3_CB1024_CMT_H1024_NRES3/meta \
    --data_root ./dataset/HumanML3D \
    --split test \
    --gpu_id 0 \
    --mode both
📊 Paper Results — Table 4, MG-MotionLLM (monocular real-world video motion dataset)
Method BLEU-1↑ BLEU-4↑ ROUGE-L↑ CIDEr↑ BERTScore↑ Avg Drop vs Ref 3D↑
Ref 3D Input 0.556 0.117 0.450 0.233 0.436 –
3D Input (TRACE) 0.385 0.033 0.310 0.058 0.240 −50.7%
3D Input (WHAM) 0.433 0.064 0.350 0.095 0.308 −35.6%
2D Input 0.392 0.044 0.321 0.062 0.264 −46.7%
3D Input (WHAM) w/ A_3D 0.469 0.036 0.378 0.106 0.352 −34.9%
2D Input w/ A_real 0.507 0.093 0.411 0.151 0.374 −17.5%

🎯 Training

Three steps: (1) train the 2D motion encoder E_2D, (2) train the real-video adapter A_real for ViTPose inputs, (3) train the adapter A_3D for estimated-3D inputs (the fair-comparison arm).

[!TIP] The 2D interface requires no training for 2DTM2T. It reuses E_2D and the adapters trained in 2DMotionGPT/ — which is the whole point of the plug-and-play design. (The TM2T base model itself was retrained once with the matched VQ-VAE architecture; see Repository Layout.)

Step 1: Train 2D Encoder

2DMotionGPT (encoder-only; also used by 2DTM2T):

cd 2DMotionGPT
python train_2d_vqvae_ver4_coco_normalized.py \
    --cfg configs/config_h3d_stage1.yaml --nodebug

2DMG-MotionLLM (trains its own 2D encoder independently):

cd 2DMG-MotionLLM
python 2d_vq_train.py --dataname t2m

Step 2: Train 2D Adapter (ViTPose → latent space)

2DMotionGPT (adapter also used by 2DTM2T):

cd 2DMotionGPT
python -m train_adapter \
    --cfg configs/config_h3d_stage1.yaml --nodebug \
    --vqvae_ckpt ./checkpoints/2d_vqvae_ver3/MotionGPT_2DEncoder/encoder_only-l1-bs64-lr0.0001/best_vqvae_epoch2960_valacc0.4471.tar \
    --estimated_motion_dir ./dataset/humanml3d_for_render

2DMG-MotionLLM:


cd 2DMG-MotionLLM
python -u train_adapter.py \
    --vqvae_2d_ckpt ./checkpoints/2d_vq_train/t2m/best_2dvq_epoch1881_ratio0.5557.pt \
    --estimated_motion_dir ../2DMotionGPT/dataset/humanml3d_for_render \
    --meta_dir ../2DMotionGPT/deps/t2m/t2m/VQVAEV3_CB1024_CMT_H1024_NRES3/meta \
    --data_root ./dataset/HumanML3D \
    --ckpt_save_dir ./checkpoints/adapter/MG-MotionLLM \
    --max_epochs 3000 --batch_size 64 --val_every 10 \
    --adapter_type residual \
    --wandb_project MG-MotionLLM-Adapter \
    --gpu_id 7

Step 3: Train 3D Adapter (WHAM → latent space)

2DMotionGPT (adapter also used by 2DTM2T):

cd 2DMotionGPT
python train_adapter_3d.py \
    --cfg configs/config_h3d_stage1.yaml --nodebug \
    --estimated_motion_dir ./dataset/humanml3d_for_render_wham

2DMG-MotionLLM:

cd 2DMG-MotionLLM
python train_adapter_3d.py \
    --estimated_motion_dir ../2DMotionGPT/dataset/humanml3d_for_render_wham \
    --val_every 10 \
    --val_text_every 100

🚧 Limitations

  • Over-generalized captions. On real videos, MotionGPT and MG-MotionLLM show a larger Avg Drop vs Ref 3D than TM2T. A likely cause is that newer MoLMs generate more detailed descriptions, while the adapter's correction smooths the motion sequence and suppresses fine-grained differences, pushing the representation toward an average motion pattern. Decoding the 2D-derived tokens back through the VQ-VAE decoder confirms this: with the adapter, an S-shaped walking trajectory is no longer preserved in the reconstructed root trajectory.
  • Imitation error in the dataset. Because videos were collected by asking participants to imitate HumanML3D motions, discrepancies between the reference motion and the performed motion can cause semantic mismatches (86.4% of pairs were verified as consistent).
  • Scope of the 2D-vs-3D conclusion. The comparison holds under the evaluated monocular pose-estimation setting (TRACE, WHAM, ViTPose), and the analysis is conducted where 2D features are explicitly aligned to the 3D latent space. It does not establish that 3D-specific information is unnecessary in general.

📖 Citation

If you find this work useful, please consider citing:

@inproceedings{yokoyama2026plugandplay,
  title     = {A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models},
  author    = {Yokoyama, Kaname and Ukita, Norimichi},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV) Workshops,
               Human-Centric Motion Intelligence Workshop (HCMIW)},
  note      = {Oral Presentation},
  year      = {2026}
}

🙏 Acknowledgements

This work builds on the following excellent projects — we thank the authors for releasing their code and data:

  • HumanML3D — motion-language dataset
  • AMASS — source 3D motions for pseudo-real adapter training
  • TM2T — base model
  • MotionGPT — base model
  • MG-MotionLLM — base model
  • FineMotion — detailed motion descriptions
  • ViTPose — 2D pose estimation
  • WHAM · TRACE — 3D pose estimation baselines