speediedan/finetuning-scheduler
speediedan/finetuning-scheduler has 68 GitHub stars and 9 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-16 03:16 UTC. Last push 2026-11-03.
Gemessener Sternzuwachs
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- Aktualisierung verzögert
- in 1 month
- Aktualisierung verzögert
Nettoänderungen in überlappenden Zeiträumen, keine Tagessummen. Fehlende Messwerte bleiben leer. Sterne zeigen Interesse, nicht Qualität.
Über uns
A PyTorch Lightning extension that accelerates and enhances foundation model experimentation with flexible fine-tuning schedules.
Maschinelles LernenREADME.md
Original
A PyTorch Lightning extension that enhances model experimentation with flexible fine-tuning schedules.

FinetuningScheduler is simple to use yet powerful, offering a number of features that facilitate model research and exploration:
- easy specification of flexible fine-tuning schedules with explicit or regex-based parameter selection
- implicit schedules for initial/naive model exploration
- explicit schedules for performance tuning, fine-grained behavioral experimentation and computational efficiency
- automatic restoration of best per-phase checkpoints driven by iterative application of early-stopping criteria to each fine-tuning phase
- composition of early-stopping and manually-set epoch-driven fine-tuning phase transitions
- an experimental PyTorch API (TuningScheduler) that brings the same schedules to a plain PyTorch training loop, without Lightning
Setup
Step 0: Install from PyPI
uv is the preferred installation approach for Fine-Tuning Scheduler.
# Install uv if you haven't already (one-time setup)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Fine-Tuning Scheduler
uv pip install finetuning-scheduler
Additional installation options
Install Optional Packages
To install additional packages required for examples:
uv pip install finetuning-scheduler['examples']
or to include packages for examples, development and testing:
uv pip install finetuning-scheduler['all']
Source Installation Examples
Using the build script (recommended):
The build_fts_env.sh script automatically handles Lightning commit pinning and optional PyTorch nightly installation:
git clone https://github.com/speediedan/finetuning-scheduler.git
cd finetuning-scheduler
# Standard development build (handles Lightning pin automatically and installs an optional PyTorch prerelease if configured in `requirements/ci/torch-pre.txt`)
./scripts/build_fts_env.sh --repo-home=${PWD} --target-env-name=fts_latest --venv-dir=/path/to/.venvs
# To configure PyTorch prerelease used by the build scripts, edit `requirements/ci/torch-pre.txt`:
# Line 1: torch version (e.g., 2.15.0 for test/RC or 2.16.0.dev20261007 for nightly)
# Line 2: CUDA target (e.g., cu132) — CI uses cpu
# Line 3: channel type: "test" or "nightly"
# Note: `manage_standalone_processes.sh` is an optional wrapper to run long-running coverage/build scripts with checks for other concurrent conflicting processes, and run options (--use-nohup etc) — you can also run `build_fts_env.sh` and `gen_fts_coverage.sh` directly.
# Activate (use your venv base path)
export FTS_VENV_BASE=~/.venvs # or /mnt/cache/${USER}/.venvs
export FTS_TARGET_VENV=fts_latest
source ${FTS_VENV_BASE}/${FTS_TARGET_VENV}/bin/activate
Manual installation with Lightning commit pin:
git clone https://github.com/speediedan/finetuning-scheduler.git
cd finetuning-scheduler
# Set UV_OVERRIDE to use the pinned Lightning commit
export UV_OVERRIDE=${PWD}/requirements/ci/overrides.txt
uv pip install -e ".[all]"
Manual installation with a PyTorch prerelease (nightly/test)
When using a PyTorch prerelease (nightly or test), use a two-step installation approach:
git clone https://github.com/speediedan/finetuning-scheduler.git
cd finetuning-scheduler
# Step 1: Install a PyTorch prerelease (adjust version and CUDA target as needed; see configuration in requirements/ci/torch-pre.txt)
# Example (test/RC channel):
uv pip install --prerelease=if-necessary-or-explicit torch==2.15.0 --index-url https://download.pytorch.org/whl/test/cu132
# Step 2: Install FTS with Lightning commit pin (torch already installed, will be skipped)
export UV_OVERRIDE=${PWD}/requirements/ci/overrides.txt
uv pip install -e ".[all]"
The prerelease version is configured via requirements/ci/torch-pre.txt. The lock_ci_requirements.sh and build_fts_env.sh scripts will read this file to determine whether to pre-install a prerelease PyTorch during builds.
Install a specific FTS version from source using the standalone pytorch-lighting package:
export FTS_VERSION=2.14.1
export PACKAGE_NAME=pytorch
git clone -b v${FTS_VERSION} https://github.com/speediedan/finetuning-scheduler
cd finetuning-scheduler
export UV_OVERRIDE=${PWD}/requirements/ci/overrides.txt
uv pip install -e ".[all]"
Latest Docker Image
Note, publishing of new finetuning-scheduler version-specific docker images was paused after the 2.0.2 patch release. If new version-specific images are required, please raise an issue.
Step 1: Import the FinetuningScheduler callback and start fine-tuning!
import lightning as L
from finetuning_scheduler import FinetuningScheduler
trainer = L.Trainer(callbacks=[FinetuningScheduler()])
Get started by following the Fine-Tuning Scheduler introduction which includes a CLI-based example or by following the notebook-based Fine-Tuning Scheduler tutorial.
Installation Using the Standalone pytorch-lightning Package
applicable to versions >= 2.0.0
Now that the core Lightning package is lightning rather than pytorch-lightning, Fine-Tuning Scheduler (FTS) by default depends upon the lightning package rather than the standalone pytorch-lightning. If you would like to continue to use FTS with the standalone pytorch-lightning package instead, you can still do so as follows:
Install a given FTS release (for example v2.0.0) using standalone pytorch-lightning:
export FTS_VERSION=2.0.0
export PACKAGE_NAME=pytorch
wget https://github.com/speediedan/finetuning-scheduler/releases/download/v${FTS_VERSION}/finetuning-scheduler-${FTS_VERSION}.tar.gz
uv pip install finetuning-scheduler-${FTS_VERSION}.tar.gz
Dynamic Versioning
FTS enables dynamic versioning both at installation time and via CLI post-installation. Initially, the dynamic versioning system allows toggling between Lightning unified and standalone imports. The two conversion operations are individually idempotent and mutually reversible.
Toggling Between Unified and Standalone Lightning Imports
FTS provides a simple CLI tool to easily toggle between unified and standalone import installation versions post-installation:
# Toggle from unified to standalone Lightning imports
toggle-lightning-mode --mode standalone
# Toggle from standalone to unified Lightning imports (default)
toggle-lightning-mode --mode unified
Note: If you have the standalone package (
pytorch-lightning) installed but not the unified package (lightning), toggling to unified mode will be prevented. You must install thelightningpackage first before toggling.
This can be useful when:
- You need to adapt existing code to work with a different Lightning package
- You're switching between projects using different Lightning import styles
- You want to test compatibility with both import styles
Examples
Scheduled Fine-Tuning For SuperGLUE
- Notebook-based Tutorial
- PyTorch API Notebook-based Tutorial (experimental, new in 2.15)
- CLI-based Tutorial
- FSDP Scheduled Fine-Tuning
- LR Scheduler Reinitialization (advanced)
- Optimizer Reinitialization (advanced)
Continuous Integration
Fine-Tuning Scheduler is rigorously tested across multiple CPUs, GPUs and against major Python and PyTorch versions.
Versioning Policy: Fine-Tuning Scheduler minor releases are aligned with core PyTorch minor releases rather than Lightning releases. This approach:
- Provides greater flexibility to integrate the latest PyTorch functionality increasingly important in research
- Reduces maintenance burden while continuing to support the stable Lightning API and robust integration
- Officially supports at least the latest 4 PyTorch minor releases (e.g., when PyTorch 2.13 is released, FTS supports >= 2.10)
This versioning approach is motivated by Lightning's evolving release cadence (see Lightning Issue #21073 and PR #21107) and allows FTS to adopt new PyTorch capabilities more rapidly while maintaining clear deprecation policies.
See the versioning documentation for complete details on compatibility policies and migration guidance.
Current build statuses for Fine-Tuning Scheduler
| System / (PyTorch/Python ver) | 2.12.0/3.11 | 2.15.0/3.11, 2.15.0/3.13 |
|---|---|---|
| Linux [GPUs**] | - | |
| Linux (Ubuntu 22.04) | ||
| OSX (14) | ||
| Windows (2022) |
- The left column is the oldest supported PyTorch version, the right column the latest tested. PyTorch
2.12.0is only exercised with Python3.11; the latest is exercised with both3.11and3.13. - The
mainbranch tracks the PyTorch 2.15 release candidates (thetestchannel), so the latest column is currently an RC rather than a published release. Released FTS versions always pin a stable PyTorch. See the compatibility table. - All GitHub Actions legs above are CPU-only (no CUDA).
- ** Multi-GPU tests run on Azure Pipelines against PyTorch
2.15.0(RC) / CUDA13.2(toolkit13.2.1) / Python3.13, on one RTX 4090 and one RTX 2070 SUPER.
Community
Fine-Tuning Scheduler is developed and maintained by the community in close communication with the Lightning team. Thanks to everyone in the community for their tireless effort building and improving the immensely useful core Lightning project.
PR's welcome! Please see the contributing guidelines (which are essentially the same as Lightning's).
Citing Fine-Tuning Scheduler
Please cite:
@misc{Dan_Dale_2022_6463952,
author = {Dan Dale},
title = {{Fine-Tuning Scheduler}},
month = Feb,
year = 2022,
doi = {10.5281/zenodo.6463952},
publisher = {Zenodo},
url = {https://zenodo.org/record/6463952}
}
Feel free to star the repo as well if you find it useful or interesting. Thanks 😊!