RegiaYoung/SlideFormer
RegiaYoung/SlideFormer has 138 GitHub stars and 5 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 18:51 UTC. Last push 2026-10-28.
What you can do
- Layer-sliding execution: keeps only a compact active layer window on the GPU, allowing full-parameter fine-tuning of models that exceed GPU memory capacity.
- Lightweight asynchronous engine: overlaps GPU computation with CPU optimizer updates, FP32-to-BF16 conversion and H2D parameter transfers, D2H gradient movement, activation offload/prefetch, and optional NVMe I/O.
- Heterogeneous memory hierarchy: coordinates GPU memory, CPU RAM, and optional local NVMe storage to reduce memory pressure while preserving full-parameter mixed-precision training semantics.
- Memory-efficient implementation: uses pre-allocated GPU cache units, layer-shared host buffers, and a layer-wise CPU LayerAdam path to reduce allocation overhead and peak GPU/CPU memory usage.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU (DAC ‘26)
AIREADME.md
OriginalSlideFormer
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU
Installation | Quick Start | Benchmarking | Supported Models | Citation
SlideFormer is a PyTorch-based heterogeneous runtime for full-parameter fine-tuning on a single GPU. It co-designs layer-sliding execution, heterogeneous memory management, and asynchronous transfer/compute pipelines across GPU memory, CPU RAM, and optional NVMe storage. Only a compact active layer window is materialized on the GPU, while persistent training states are kept in CPU memory and cross-tier data movement is carefully scheduled throughout execution.
In our evaluation, SlideFormer achieves 1.40×–6.27× higher throughput than related offloading baselines, reduces peak GPU memory by up to 50% and peak CPU memory by up to 40%, supports up to 8× larger batch sizes, and enables full-parameter fine-tuning of 100B+ models on a single commodity GPU.
If this repository is useful to your work, please consider starring it.
News
- 2026.09 — Multi-GPU extension: A multi-GPU version of SlideFormer is
now available in
multi_gpu/. For our latest research on multi-GPU LLM fine-tuning, see the SlideDP repository and arXiv paper. - 2026.06: We released SlideFormer with expanded model compatibility, reproducibility scripts, and a new fine-grained chunked pipeline.
- 2026.03: SlideFormer was accepted to DAC 2026. The conference presentation page is available here, and the paper is available on arXiv.
- Early 2025: The initial SlideFormer system was developed and evaluated; subsequent development has focused on system-level optimization.
Highlights
Layer-sliding execution: keeps only a compact active layer window on the GPU, allowing full-parameter fine-tuning of models that exceed GPU memory capacity.
Lightweight asynchronous engine: overlaps GPU computation with CPU optimizer updates, FP32-to-BF16 conversion and H2D parameter transfers, D2H gradient movement, activation offload/prefetch, and optional NVMe I/O.
Heterogeneous memory hierarchy: coordinates GPU memory, CPU RAM, and optional local NVMe storage to reduce memory pressure while preserving full-parameter mixed-precision training semantics.
Memory-efficient implementation: uses pre-allocated GPU cache units, layer-shared host buffers, and a layer-wise CPU LayerAdam path to reduce allocation overhead and peak GPU/CPU memory usage.
Installation
SlideFormer is designed for single-GPU machines with sufficient CPU memory. The exact memory requirement depends on the model size, sequence length, batch size, optimizer state, and offload configuration.
From our evaluation sweep, a practical estimate is about 12 GB CPU RAM per 1B parameters + batch-dependent activation/buffer overhead. NVMe optimizer/activation offload can further reduce the CPU-memory requirement at the cost of lower throughput.
git clone https://github.com/RegiaYoung/SlideFormer.git
cd SlideFormer
conda env create -f environment.yml
conda activate slideformer
For newer models such as Qwen3.6 and Gemma 4, we recommend using the updated
dependencies in requirements-models.txt:
pip install packaging ninja==1.13.0
pip install --no-build-isolation -r requirements-models.txt
Core dependencies are
torch,
transformers, and
datasets. Since the core
runtime is mostly torch-native, SlideFormer can also run on AMD/ROCm GPUs with a
compatible PyTorch/ROCm stack.
Optional dependencies for acceleration/offload are
flash-attn,
liger-kernel, and
tensornvme; the code falls back
when they are not available. We recommend installing all optional dependencies
to match the reported performance.
Quick Start
Run a dummy-data end-to-end sanity check:
python scripts/main_dummy.py \
--model_path /path/to/Llama-3.1-8B-Instruct \
--seq_len 1024 \
--batch_size 64
On multi-NUMA machines, binding the process to the GPU's NUMA node can improve CPU-GPU bandwidth:
numactl --cpunodebind=0 --membind=0 python scripts/main_dummy.py
Useful options:
| Option | Description |
|---|---|
--model_path |
Local path or HuggingFace model ID |
--seq_len |
Sequence length |
--batch_size |
Per-step batch size |
--epochs |
Number of training epochs |
--attn_implementation |
flash_attention_2 or sdpa |
--ac_offload_nvme |
Offload saved activations to NVMe |
--nvme_offload_fraction |
Optimizer-state NVMe offload fraction |
--offload_dir |
Directory used when NVMe offload is enabled |
Real Data Example
scripts/main_real.py provides a real fine-tuning example on
QizhiPei/MathFusionQA.
python scripts/main_real.py \
--model_path /path/to/Llama-3.1-8B-Instruct \
--seq_len 4096 \
--batch_size 16 \
--epochs 1 \
--output_dir ./mathfusion-ft-results
The script saves the trained model, tokenizer, loss curve, and loss CSV to
--output_dir.
Correctness Validation
SlideFormer tracks a DeepSpeed ZeRO-3 CPU-offload reference on both 40 distinct real-data batches and a repeated-batch overfit probe.

Benchmarking
Single benchmark run:
python scripts/main_bench.py \
--model_path /path/to/Llama-3.1-8B-Instruct \
--seq_len 1024 \
--batch_size 64 \
--warm_step 3 \
--test_step 10 \
--result_file ./outputs/bench.csv
Sweep over configured models and batch sizes:
bash scripts/bench.sh
For reproducibility, baseline implementations and adapted configs are collected
under bench/, including DeepSpeed,
ColossalAI,
MegaTrain, and
LoHan. The supplemental figures below add a
MegaTrain comparison alongside the paper baselines.
![]() (a) Batch-size scaling and CPU memory for Llama-3.1-8B on RTX 4090. |
![]() (b) Model-size scaling and CPU memory for Qwen3 models on RTX 4090. |
![]() (c) GPU allocated memory vs. batch size for Llama-3.1-8B. |

(d) Long-context training on RTX 4090. Left: sequence-length scaling for Llama-3.1-8B at batch size 1. Right: maximum trainable sequence length for Qwen3 models at batch size 1; labels above bars report TFLOPS at the corresponding maximum.
With the flash attention used here, activation-related memory scales
approximately with batch_size * seq_len. For example, SlideFormer shows nearly
identical CPU and allocated GPU memory at bs=64, seq=1K and
bs=1, seq=64K. Compute scales differently: self-attention costs
O(B * S^2), so increasing sequence length adds substantially more attention
work even at a fixed token budget and makes training more compute-intensive.
Long-context TFLOPS can consequently exceed the 1K no-offload peak, which is
a workload-specific reference rather than a hardware peak.
Note: For fairness, all evaluated systems use the same workload and enable the same fused kernels unless a framework already provides an equivalent implementation. SlideFormer and all baselines except MegaTrain follow mixed-precision training semantics with BF16 compute, FP32 master parameters, and FP32 optimizer states. In the official MegaTrain single-GPU path, CPU-resident model parameters are loaded and updated in BF16. Its CPU-memory footprints are therefore shown for reference only.
Technical Updates
Multi-GPU Extension
We provide a reference multi-GPU extension of SlideFormer in
multi_gpu/. It preserves SlideFormer's layer-streaming
execution while adding synchronous data parallelism with replicated parameter
delivery and CPU-side gradient reduction. See
multi_gpu/README.md for setup and usage.
For optimized shared-host multi-GPU execution, follow SlideDP and read the paper. The optimized SlideDP runtime described in the paper is not included in this SlideFormer release.
Fine-grained Chunked Overlap
The 2026.06 release adds a chunked asynchronous transfer/update pipeline that splits long FP32-to-BF16 conversion, H2D parameter movement, D2H gradient return, and CPU Adam update into smaller overlapped segments. This reduces exposed transfer/update time under pipeline contention and improves Qwen3-8B single-GPU throughput, especially at small batch sizes.

Supported Models
- Qwen2, Qwen2.5, Qwen3, and Qwen3.6 families
- Llama 3, 3.1, 3.2, and 3.3 families
- Mistral family
- Gemma 4 family: 26B-A4B and 31B.
We support full-parameter fine-tuning across a wide range of Hugging Face language models. Support for multimodal models is under active development.
Repository Layout
SlideFormer/
├── offload_transformer.py # Runtime engine and scheduling
├── transformer_layer.py # Layer wrappers and CPU-GPU transfers
├── sliding_checkpoint.py # Activation offload and prefetch
├── multi_gpu/ # SlideFormer multi-GPU version
├── optimizer/ # Layer-wise CPU Adam optimizer
├── utils/ # Datasets, metrics, and monitor helpers
├── scripts/ # Train, bench, and profile entries
├── bench/ # Baselines and reproducibility artifacts
└── environment.yml # Conda environment
Citation
If you use SlideFormer, its code, or its design ideas in your research, please cite our DAC 2026 paper.
- Published paper: ACM Digital Library
- Author version: arXiv:2603.16428
- DAC 2026 presentation: 63rd ACM/IEEE Design Automation Conference
- DOI: 10.1145/3770743.3804125
- ISBN:
979-8-4007-2254-7
@inproceedings{yang2026efficientheterogeneouscodesignfinetuning,
author = {Yang, Ruijia and Wen, Zeyi},
title = {An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single {GPU}},
booktitle = {Proceedings of the 63rd ACM/IEEE Design Automation Conference},
series = {DAC '26},
year = {2026},
publisher = {Association for Computing Machinery},
isbn = {9798400722547},
articleno = {472},
pages = {1--7},
numpages = {7},
doi = {10.1145/3770743.3804125},
url = {https://dl.acm.org/doi/10.1145/3770743.3804125}
}
License
This project is released under the Apache-2.0 License. See NOTICE for attribution and bundled third-party software information.


