Lowaltitudeflight/local-llm-tuning-guide
Lowaltitudeflight/local-llm-tuning-guide has 119 GitHub stars and 0 forks. GitHot measured +5 stars in 24 hours. Last checked 2026-11-18 08:45 UTC. Last push 2026-11-12.
Measured star growth
- in 24h
- +5
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.
AIREADME.md
OriginalEnglish | 简体中文
Local LLM Tuning Guide
Decisions, not theory, for multi-GPU LLM serving on PCIe machines without NVLink. Every claim carries an evidence grade and the conditions under which it holds.

| What you get | Detail |
|---|---|
| Measured, not repeated | 8 datasets, three independent launches each, per-configuration variance |
| Failures included | 9 failed configurations — what never starts, and what runs but misleads |
| Numbers you can check | every KV figure re-derived in CI; every dataset declares its reproducibility status |
| Bilingual | every chapter and recipe in English and 简体中文 |
Three findings that did not match the folklore:
- TP8 used 2× the cards for +12% throughput — per GPU, TP4 was 1.7× better.
- A 476 GiB checkpoint cannot run TP4 on 4×72 GB — it OOMs at 70.74 GiB, 48 GiB short per card.
- KV cache measured 785 B/token, 30× smaller than the GQA formula predicts for this model.
Why this exists
Most self-hosted inference today runs on machines without NVLink: RTX 4090 / 5090, RTX PRO 6000, PCIe variants of datacenter cards, connected over PCIe 5.0 x16 or a PCIe switch.
Yet the documentation, tuning advice, and default configurations of mainstream inference frameworks are almost all written under the assumption that GPUs talk over a fast interconnect. The consequences:
- The same four GPUs can differ by more than 2× in throughput (community reports; see benchmarks) — the difference is configuration, not silicon
- Articles explaining the principles are easy to find, but their parameter recommendations contradict each other and rarely state test conditions
- You cannot tell whether a piece of advice applies to your machine
This project supplies the missing half: binding each conclusion to the conditions under which it is valid.
Who this is for
- You run 2–8 GPUs on one box, over PCIe, without NVLink.
- You are choosing TP / PP / EP / replicas, quantization, or KV settings and want a starting point with its evidence attached.
- You have a benchmark number and need to know whether it transfers to your machine.
Not for: multi-node fabrics (InfiniBand, NVSwitch), training, or “which model is smartest” rankings. The guide is about serving configuration, not model quality.
Start here
- Run
python tools/detect_topology.pyand keep the JSON — it is the input to every later decision. - Open the model matrix to see whether your model fits, and on how many cards.
- Pick the closest recipe, copy the flags, then measure with bench/run_matrix.py instead of trusting the defaults.
Quick decision table
Six highest-frequency questions, answered first.
| Your situation | Recommendation | Basis |
|---|---|---|
| Model (plus KV) fits on one GPU | Prefer replicas over tensor parallelism | Measured + derivable |
| Model must be split; all GPUs under one PCIe switch | Start with TP = GPUs under that switch, then measure. On the 8×72 box TP8 beat TP4-on-one-node by 12% aggregate — but it used 2× the cards; TP4 was 1.7× better per GPU | Measured |
| GPUs span PCIe switches or CPU sockets | Avoid letting TP cross that boundary by default — use TP × PP or replicas. The same box measured one exception (TP8 across SYS still beat TP4-on-node); treat this as a default, not a law | Derivable + measured |
| You want a longer context | Lower KV precision or add GPUs first; DCP gains little over PCIe | Measured + derivable |
| You want speculative decoding for speed | Longer drafts are not automatically better — start at 2–3. The published 19–22% gap compares MTP3 with DFlash2-K7, which differ in draft method too | Measured (confounded) |
| Quantization hurts output quality | Raise activation precision first — W4A4 → W4A16 took checksum failures from 5.9% to 0. FP32 router weights are a free second safeguard | Measured |
Contents
| Chapter | Topic | In one line |
|---|---|---|
| 01 Hardware & topology | The real gap between PCIe and NVLink | Find out how your GPUs are actually connected |
| 02 Parallelism | TP / PP / DP / EP / DCP | When adding a GPU makes things slower |
| 03 Communication | All-reduce modes and NCCL tuning | Why PCIe prefers one-shot |
| 04 Quantization | FP8 / MXFP8 / NVFP4 / AWQ trade-offs | What you pay for the memory you save |
| 05 Speculative decoding | MTP / DFlash / EAGLE families | Why guessing more can be slower |
| 06 KV cache | Memory budget, prefix cache, long context | How context eats your VRAM |
| 07 Troubleshooting | Symptom → cause → fix | OOM, bad output, SM120 kernel gaps, hung containers |
| 08 Power & cost | Tokens per watt and per dollar | Why a 300 W card stops scaling |
| 09 Failed configurations | What does not work, and why | The negative half of the library |
| Model matrix | What fits on 72 GB cards | Weights, KV/token, minimum GPU count, verified configs |
| Recipes | Ready-to-use configs by hardware | 4×72 · 8×72 · 4×96 · 8×24 |
| Benchmarks | Data tables and measurement method | Fill the table with your own machine |
| Planner | Hardware in, parallel strategy out | CLI + offline web tool · try the web tool online |
| Glossary | Terms and abbreviations | TP / PP / EP / MLA / NVFP4 / SM120 |
| Articles | Long-form write-ups | English abstracts + Chinese full text |
Every chapter is also available in 简体中文 — as are the recipes, the benchmark pages, and CONTRIBUTING.
Evidence grading
Every conclusion carries a basis tag:
| Tag | Meaning |
|---|---|
| Measured | Produced by a run on real hardware. Hardware, software versions and variance are recorded; new submissions must also record clock policy, image digest and the exact server/client commands (see SCHEMA). Entries submitted before 2026-11 lack the last three and are marked as such — they were run, but are not yet fully reproducible |
| Derivable | Calculated from a model config, hardware specification or framework mechanic; independent of any single measurement. In tables, a derivable figure says so (for example the KV/token derivation block in MODEL_MATRIX) |
| Open | Direction is clear but public data is missing; we provide a measurement method and leave the cell to be filled |
What this project does not do: invent numbers. An empty cell is worth more than a fabricated one.
Key conclusions at a glance
Six of the highest-confidence judgements, expanded in the chapters.
- If it fits on one GPU, do not use tensor parallelism. Two independent replicas have zero inter-GPU communication and scale throughput linearly; TP all-reduces at every layer, and over PCIe that cost eats most of the gain.
- The boundary for TP is the PCIe switch, not the GPU count. Traffic crossing switches goes through CPU memory with a sharp latency penalty. With four GPUs split across two switches, TP=4 is often worse than TP=2 × PP=2 — sometimes worse than two TP=2 replicas.
- Over PCIe, prefer one-shot all-reduce. The two-shot scheme is superior on a fast interconnect, but that superiority assumes ample bandwidth. PCIe is the opposite case.
- Speculative decoding gains are bandwidth-bound. A longer draft wastes more compute per failed verification, and the draft model itself consumes bandwidth. The published comparison (MTP3 vs DFlash2-K7) shows the shorter draft ahead by 19–22%, but the two configurations also differ in draft method and weights — read it as "measure both", not "7 is always worse than 3".
- Raise activation precision before touching the router. In MoE models, W4A4 → W4A16 alone took greedy-checksum failures from 5.9% to 0. Keeping FP32 router weights costs no measurable throughput and is the difference on pre-QAD checkpoints (19.5% → 4.0%), but it does not substitute for A16 activations.
- On one machine, configuration alone accounts for more than a 2× throughput difference. This is not folklore — every parameter is either aligned with your real topology or fighting it.
How to use this
Deploying a new machine: start with 01 Hardware & topology, run nvidia-smi topo -m once, then find the closest match in recipes.
Machine already running but slow: run the planner and diff its parameters against your current ones.
python tools/planner.py --gpus A800-80 --count 8 --model qwen2.5-72b --ctx 8192
It reports the interconnect ceiling, the memory budget, and a TP / PP / replica layout — each tagged with an evidence grade. See tools/README.md.
Want to contribute data: see CONTRIBUTING.md. The tables are ready; what is missing is measurements from real machines.
Related projects
- llm-inference-bench — the companion benchmarking platform that produces the data this project consumes
- llm-private-deploy-handbook — delivery process handbook for private deployments