Skip to content
Proyectos

traceopt-ai/traceml has 189 GitHub stars and 32 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and 0 in 15 days. Last checked 2026-11-18 03:10 UTC. Last push 2026-11-14.

Cómo se miden estas cifras

Crecimiento de estrellas medido

in 24h
0
in 7 days
0
in 15 days
0
in 1 month
Recopilando historial

Cambios netos en períodos que se solapan, no totales diarios. Los datos no disponibles se dejan en blanco. Las estrellas indican interés, no calidad.

Acerca de

Find why PyTorch training is slow, compare runs, and catch performance regressions in CI.

Aprendizaje automático

README.md

Original

TraceML

Diagnose slow PyTorch training. Compare runs and catch performance regressions.

Works automatically with: Hugging Face Trainer · PyTorch Lightning · RF-DETR

PyPI version CI Python 3.10+ License GitHub stars

Quickstart · Integrations · Distributed training · Monitoring · Compare runs · Documentation

⭐ If TraceML helps you find a bottleneck, please star the repository.

TraceML breaks down training-step time, identifies likely bottlenecks, and saves reports for comparison or CI.

  • Trainer integrations: automatic setup for supported trainers; explicit step instrumentation for custom PyTorch loops.
  • Distributed diagnosis: compare rank timings and identify stragglers across GPUs and nodes.
  • Continuous monitoring: collect timing and resource metrics throughout training.

Quickstart

1. Install

If your training framework is already installed:

pip install traceml-ai

2. Run your existing script

- python train.py
+ traceml run train.py

For the standard trainer APIs listed above, no script changes are needed. TraceML prints a diagnosis when training finishes and writes final_summary.json and final_summary.txt under logs/<run-name>/.

No training script ready? Try the interactive demo, the Colab example, or the RF-DETR sample-data demo, which needs no dataset download.

What you get

The terminal report identifies the likely bottleneck, shows the timing and resource evidence behind it, and suggests the next investigation.

+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|  TraceML Run Summary                                                                                                                                     |
|  bert_finetune · 1 rank · 1 GPU observed · 256 common steps · 52.4s                                                                                      |
+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                                                          |
|  Verdict: INPUT-BOUND  (CRITICAL)                                                                                                                        |
|  Why: Input Wait took 64% of Step Time.                                                                                                                  |
|  Next: Increase workers, prefetch, or storage throughput.                                                                                                |
|                                                                                                                                                          |
|  STEP TIMING (Window Average), GPU Clock                      ||  STEP MEMORY: BALANCED                                                                  |
|  Step Time           200.4 ms  100%                           ||                                                                                         |
|  ├─ Input Wait       128.0 ms   64%  ◀  cause                 ||                                                                                         |
|  ├─ Compute           68.0 ms   34%                           ||  avg per-step peak           avg                                                        |
|  │  ├─ Forward        24.0 ms   12%                           ||  Allocated                   2.9 GB                                                     |
|  │  ├─ Backward       38.0 ms   19%                           ||  Reserved                    3.2 GB                                                     |
|  │  └─ Optimizer       6.0 ms    3%                           ||                                                                                         |
|  ├─ H2D                0.4 ms   <1%                           ||                                                                                         |
|  └─ Residual           3.6 ms    2%                           ||                                                                                         |
|  DataLoader fetch: 120.0 ms (CPU, supplemental)               ||                                                                                         |
|                                                                                                                                                          |
|  SYSTEM METRICS: LOW GPU UTIL                                 ||  PROCESS METRICS: NORMAL                                                                |
|  Evidence: GPU utilization averaged 24%.                      ||                                                                                         |
|                                                               ||                                                                                         |
|                         avg                                   ||                       avg                                                               |
|  CPU                    18%                                   ||  CPU capacity         14%                                                               |
|  RAM used               6.2 GB (19%)                          ||  RSS used             3.1 GB (10%)                                                      |
|  GPU util               24%                                   ||  CUDA allocated       2.9 GB                                                            |
|  GPU memory/device      3.3 GB (21%)                          ||  CUDA reserved        3.2 GB (20%)                                                      |
|  GPU temperature        42C                                   ||                                                                                         |
|  GPU power              58W                                   ||                                                                                         |
|                                                                                                                                                          |
|                                                                                                                                                          |
|  Full evidence: logs/bert_finetune/final_summary.json  (--html-report)                                                                                   |
+----------------------------------------------------------------------------------------------------------------------------------------------------------+

What TraceML diagnoses

Diagnosis Where to investigate
Input-bound DataLoader workers, transforms, tokenization, collation, or storage
H2D-bound Pinned memory, non-blocking copies, batch size, or transfer overlap
Compute-bound Model compute, mixed precision, batch size, or deeper profiling
Residual-heavy Logging, checkpointing, validation, CPU stalls, or unobserved work
Rank straggler Rank-local input, data imbalance, node variance, or networking
Memory creep Retained tensors, logging references, or cached activations

Read How to Read TraceML Output for definitions, diagnosis rules, and evidence limits.

Integrations

For these standard trainer APIs, traceml run needs no changes to your script:

Framework Training API
Hugging Face Trainer trainer.train()
PyTorch Lightning trainer.fit(...)
RF-DETR model.train(...)

Custom loops and other frameworks

Use the following setup for custom loops and other frameworks:

Training path Setup
Plain PyTorch or a custom loop traceml.init() and trace_step(...)
Hugging Face Accelerate Explicit step instrumentation
Ray Train and Ray Data TraceML Trainer/config wrappers
DeepSpeed Explicit step instrumentation
MONAI TraceML handler setup
Plain PyTorch example
import traceml_ai as traceml

traceml.init(mode="auto")

for batch in dataloader:
    with traceml.trace_step(model):
        optimizer.zero_grad(set_to_none=True)
        outputs = model(batch["x"])
        loss = criterion(outputs, batch["y"])
        loss.backward()
        optimizer.step()

Run it with traceml run train.py.

See the Public API and integration support matrix.

Distributed training and straggler detection

For scripts already configured for distributed training, replace torchrun with traceml run:

- torchrun --nproc-per-node=4 train.py
+ traceml run train.py --nproc-per-node=4

Across machines, run the same command on every node and change only --node-rank. On node 0:

traceml run train.py \
  --nnodes=2 \
  --node-rank=0 \
  --nproc-per-node=4 \
  --master-addr=node0 \
  --run-name=my-run \
  --logs-dir=/shared/traceml/logs

Node 0 starts the TraceML aggregator; other nodes join the same run. All nodes require a shared logs directory. Multi-node DDP uses summary mode; multi-node GPU support is experimental.

When training finishes, TraceML aligns common steps across all ranks and nodes and points to the rank most likely holding up the run:

+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|  TraceML Run Summary                                                                                                                                     |
|  ddp_pretrain · 4/4 ranks · 4 GPUs observed · 2/2 nodes · 250 common steps · 40.1s                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                                                          |
|  Verdict: INPUT STRAGGLER  (CRITICAL)                                                                                                                    |
|  Why: R0/N0 waited 254.5 ms for input; R1/N0 waited 3.8 ms for input.                                                                                    |
|  Next: Inspect input wait on the slow rank.                                                                                                              |
|  Scope: N = node · R = global rank · G = GPU index                                                                                                       |
|                                                                                                                                                          |
|  STEP TIMING (Median R1/N0), GPU Clock                        ||  STEP MEMORY: BALANCED · 4/4 ranks                                                      |
|  Step Time           303.7 ms  100%                           ||                                                                                         |
|  ├─ Input Wait         3.8 ms    1%                           ||                                                                                         |
|  ├─ Compute          259.5 ms   85%                           ||  avg per-step peak           median rank avg     worst rank avg                         |
|  │  ├─ Forward        80.0 ms   26%                           ||  Allocated                   8.5 GB              9.4 GB, R2/N1                          |
|  │  ├─ Backward      169.5 ms   56%                           ||  Reserved                    8.9 GB              9.8 GB, R2/N1                          |
|  │  └─ Optimizer      10.0 ms    3%                           ||                                                                                         |
|  ├─ H2D                1.1 ms   <1%                           ||                                                                                         |
|  └─ Residual          39.3 ms   13%                           ||                                                                                         |
|  DataLoader fetch: 3.7 ms (CPU, supplemental)                 ||                                                                                         |
|                                                                                                                                                          |
|  SYSTEM METRICS: LOW GPU UTIL · 2/2 nodes                     ||  PROCESS METRICS: NORMAL · 4/4 ranks                                                    |
|  Evidence: GPU utilization averaged 14%.                      ||                                                                                         |
|                                                               ||                                                                                         |
|                         median node avg   worst node avg      ||                       median rank avg   worst rank avg                                  |
|  CPU                    18%               26%, N1             ||  CPU capacity         12%               81%, R2/N1                                      |
|  RAM used               16.0 GB (27%)     20.8 GB (35%), N1   ||  RSS used             3.1 GB (10%)      5.4 GB (17%), R1/N0                             |
|  GPU util               9%                9%, N1              ||  CUDA allocated       2.9 GB            4.6 GB, R3/N1                                   |
|  GPU memory/device      5.0 GB (31%)      7.0 GB (44%), N1    ||  CUDA reserved        3.2 GB (20%)      6.8 GB (43%), R3/N1                             |
|  GPU temperature        58C               70C, N1             ||                                                                                         |
|  GPU power              220W              280W, N1            ||                                                                                         |
|                                                                                                                                                          |
|                                                                                                                                                          |
|  Full evidence: logs/ddp_pretrain/final_summary.json  (--html-report)                                                                                    |
+----------------------------------------------------------------------------------------------------------------------------------------------------------+

Distributed GPU comparisons assume homogeneous hardware across ranks. See Distributed Training for ports and shared storage, DDP rank stragglers, Slurm, and the integration support matrix for framework-specific details, including FSDP.

Continuous monitoring

TraceML records GPU phase timings with CUDA events and resolves them asynchronously. Telemetry is batched and processed by a separate aggregator.

Use a kernel profiler when the diagnosis points inside GPU compute.

Compare runs

After changing the DataLoader, batch size, model, or infrastructure, compare two completed runs:

traceml compare before/final_summary.json after/final_summary.json
+--------------------------------------------------------------------------------------+
|  TraceML Compare                                                                     |
+--------------------------------------------------------------------------------------+
|  A: before_dataloader_fix                                                            |
|  B: after_dataloader_fix                                                             |
|  Primary diagnosis: INPUT-BOUND -> COMPUTE-BOUND (changed)                           |
|                                                                                      |
|  Verdict: IMPROVEMENT                                                                |
|  Why: GPU Step Time decreased by 59.9%.                                              |
+--------------------------------------------------------------------------------------+

TraceML keeps the comparison evidence in JSON and text so the result is easy to review, archive, or use in automation. See Compare Runs.

Performance regression checks

Use the same comparison as a local or CI gate for compatible runs:

traceml compare \
  logs/reference/final_summary.json \
  logs/candidate/final_summary.json \
  --max-step-time-regression-pct 5 \
  --output compare/reference-vs-candidate

TraceML writes JSON and text reports before returning the CI exit code. See Regression Guard for comparability requirements and supported environments.

Featured case studies

Investigation Result
ResNet-18 input pipeline See an input-bound run become compute-bound after changing only DataLoader settings.
RF-DETR Nano training Examine single-GPU phase timing and four-GPU DDP scaling on real COCO batches.
RF-DETR release regression Attribute a release-to-release slowdown to input waiting and verify the fixed release.

Browse all case studies →

Reports and experiment trackers

Summary mode is the default. For live diagnostics in the terminal, use:

traceml run train.py --mode=cli

For the browser dashboard, install the optional dashboard dependencies:

pip install "traceml-ai[dashboard]"
traceml run train.py --mode=dashboard

TraceML can also export a self-contained HTML report and send its compact summary to an existing W&B or MLflow run. See W&B and MLflow and the complete quickstart.

Learn more

Community

Questions, contributions, and real-world slowdown reports are welcome:

License

Apache 2.0. See LICENSE.