Skip to content
Projects

NVIDIA/nvidia-resiliency-ext

View on GitHub

NVIDIA/nvidia-resiliency-ext has 356 GitHub stars and 71 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-19 02:17 UTC. Last push 2026-11-17.

How these counts are measured

What you can do

  • Section-based hang detection and coordinated worker restarts within a running job.
  • Megatron GPT recovery walkthrough with fixed CLI timeouts and NVRx-backed async checkpoints.
  • Small DDP example with section monitoring and synchronous PyTorch checkpoints.
  • Providing an efficient framework for asynchronous checkpointing.

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the effective training time by minimizing the downtime due to failures and interruptions.

Developer Tools

README.md

Original

NVIDIA Resiliency Extension

The NVIDIA Resiliency Extension (NVRx) integrates multiple resiliency-focused solutions for PyTorch-based workloads. Users can modularly integrate NVRx capabilities into their own infrastructure to maximize AI training productivity at scale. NVRx maximizes goodput by enabling system-wide health checks, quickly detecting faults at runtime and resuming training automatically. NVRx minimizes loss of work by enabling fast and frequent checkpointing.

For detailed documentation and usage information about each component, please refer to https://nvidia.github.io/nvidia-resiliency-ext/.

⚠️ NOTE: This project is still experimental and under active development. The code, features, and documentation are evolving rapidly. Please expect frequent updates and breaking changes. Contributions are welcome and we encourage you to watch for updates.

Core Components and Capabilities

Installation

From sources

  • git clone https://github.com/NVIDIA/nvidia-resiliency-ext
  • cd nvidia-resiliency-ext
  • pip install .

From PyPI wheel

  • pip install nvidia-resiliency-ext

Platform Support

Category Supported Versions / Requirements
Architecture x86_64, arm64
Operating System Ubuntu 22.04, 24.04
Python Version >= 3.11 (release wheels and CI: 3.12, 3.14)
PyTorch Version >= 2.9.0
CUDA & CUDA Toolkit >= 12.8
NVML Driver >= 535 (570 required for GPU health check)
TE Version >= 2.5