---

---

no-magic

Because model.fit() isn't an explanation.

---

What This Is

no-magic is a curated collection of single-file, dependency-free Python implementations of the algorithms that power modern AI. Each script is a complete, runnable program — no frameworks, no abstractions, no hidden complexity. Most scripts train from scratch: some train one model and then use it for inference, others train alternative variants side by side and compare them (often running inference too). A smaller set runs untrained forward-pass mechanisms or non-learning algorithms. Each script's kind is recorded explicitly (see Script contracts).

Every script in this repository is an executable proof that these algorithms are simpler than the industry makes them seem. The goal is not to replace PyTorch or TensorFlow — it's to make you dangerous enough to understand what they're doing underneath.

See It In Action

01 — Foundations (17 scripts)

Autoregressive GPT Token-by-token generation	RNN vs GRU Vanishing gradients and gating	LSTM 4-gate memory highway
BPE Tokenizer Iterative pair merging → vocabulary	Word Embeddings Contrastive learning → semantic clusters	RAG Pipeline Retrieve → augment → generate
BERT Bidirectional attention + [MASK] prediction	Convolutional Net Sliding kernels → feature maps	ResNet F(x) + x = gradient highway
Vision Transformer Image patches as tokens	Diffusion Noise → data via iterative denoising	VAE Encode → sample z → decode
GAN Generator vs discriminator minimax	Optimizers SGD vs Momentum vs Adam convergence	Error Bars for Evals CLT vs bootstrap vs Wilson coverage → paired SE → required n

Comparison programs without a preview: attention_vs_none.py · rnn_vs_gru_vs_lstm.py

02 — Alignment & Training (13 scripts)

LoRA Fine-tuning Low-rank weight injection	QLoRA 4-bit base + full-precision adapters	DPO Alignment Preferred vs. rejected → policy update
PPO (RLHF) Clipped policy gradient for alignment	GRPO Group-relative rewards, no critic	REINFORCE Log P(a) × reward = gradient
Mixture of Experts Sparse routing to specialist MLPs	Batch Normalization Normalize activations → stable training	Dropout Kill neurons → prevent overfitting
Supervised Fine-Tuning Pretrained base → masked demonstration loss	Knowledge Distillation Frozen teacher's soft targets → smaller student	ROME Knowledge Editing Key → value, one rank-one W_proj update

Comparison program without a preview: adam_vs_sgd.py

03 — Systems & Inference (19 scripts)

Attention Mechanism Q·KT → softmax → weighted V	Flash Attention Tiled O(N) memory computation	RoPE Position via rotation matrices
KV-Cache Memoize keys/values — stop recomputing	PagedAttention OS-style paged KV-cache memory	Quantization Float32 → Int8 = 4x compression
Beam Search Tree search with top-k pruning	Checkpointing O(n) → O(√n) memory via recompute	Model Parallelism Tensor + pipeline across devices
State Space Models Linear-time selective state transitions	Vector Search Exact vs LSH approximate search	BM25 TF → TF-IDF → BM25 evolution
Speculative Decoding Draft fast, verify once	Complex SSM Complex eigenvalues = real + RoPE	Discretization Euler vs ZOH vs Trapezoidal
Roofline Model SISO → MIMO hardware utilization	TurboQuant Data-oblivious quantization via random rotation	Green-List Watermarking Keyed green list → biased sampling → z-test
Self-Consistency Sample traces → parse answers → vote

04 — Agents & Planning (5 scripts)

Monte Carlo Tree Search UCB1 tree search + random rollouts	ReAct Agent Thought → Action → Observation	Multi-Armed Bandits ε-greedy vs UCB1 vs Thompson Sampling
Minimax + Alpha-Beta Adversarial search with pruning	Memory-Augmented Network Differentiable read/write heads

All algorithms have animated visualizations. Full 1080p60 videos in Releases. Visualization source and rendering: no-magic-viz — built with Manim.

Philosophy

Modern ML education has a gap. There are thousands of tutorials that teach you to call library functions, and there are academic papers full of notation. What's missing is the middle layer: the algorithm itself, expressed as readable code.

This project follows a strict set of constraints:

• One file, one algorithm. Every script is completely self-contained. No imports from local modules, no utils.py, no shared libraries.
• Zero external dependencies. Only Python's standard library. If it needs pip install, it doesn't belong here.
• A recorded lifecycle. Most scripts train a model and then use it for inference, and many trained comparisons do too; the exceptions (comparisons that stop at the comparison, untrained forward-pass demonstrations and non-learning algorithms) are recorded per script. See Script contracts and docs/catalog.json.
• Designed to run in minutes on a CPU. No GPU required. No cloud credits. Scripts are written to finish on a laptop CPU; the tier READMEs show historical timings recorded when each script was added, which are not a current measurement of every script.
• Comments are mandatory, not decorative. Every script must be readable as a guided walkthrough of the algorithm. We are not optimizing for line count — we are optimizing for understanding. See CONTRIBUTING.md for the full commenting standard.

Who This Is For

• ML engineers who use frameworks daily but want to understand the internals they rely on.
• Students transitioning from theory to practice who want to see algorithms as working code, not just equations.
• Career switchers entering ML who need intuition for what's actually happening when they call high-level APIs.
• Researchers who want minimal reference implementations to prototype ideas without framework overhead.
• Anyone who has ever stared at a library call and thought: "but what is it actually doing?"

This is not a beginner's introduction to programming. You should be comfortable reading Python and have at least a surface-level familiarity with ML concepts. The scripts will give you the depth.

What You'll Find Here

The repository is organized into four tiers based on conceptual dependency:

01 — Foundations (17 scripts)

Core algorithms that form the building blocks of modern AI systems. GPT, RNN, LSTM, BERT, CNN, ResNet, ViT, GAN, VAE, diffusion, embeddings, tokenization, RAG, and optimizer comparison, plus evaluation statistics: error bars for eval scores, paired comparison of two models and the question count an effect needs. Includes comparison scripts for attention mechanisms and recurrent architectures.

See 01-foundations/README.md for the full algorithm list, timing data, and roadmap.

02 — Alignment & Training Techniques (13 scripts)

Methods for steering, fine-tuning, and aligning models after pretraining. Supervised fine-tuning (SFT), LoRA, QLoRA, DPO, PPO, GRPO, REINFORCE, knowledge distillation, rank-one knowledge editing (ROME), MoE, batch normalization, dropout/regularization, and optimizer comparison.

See 02-alignment/README.md for the full algorithm list, timing data, and roadmap.

03 — Systems & Inference (19 scripts)

The engineering that makes models fast, small, and deployable. Attention variants, Flash Attention, KV-cache, PagedAttention, RoPE, quantization, beam search, checkpointing, parallelism, SSMs, vector search, BM25, speculative decoding, complex SSM equivalence, discretization methods, roofline analysis, green-list watermarking of generated text with its z-test detector, and self-consistency decoding over model-written reasoning traces.

See 03-systems/README.md for the full algorithm list, timing data, and roadmap.

04 — Agents & Planning (5 scripts)

Autonomous reasoning and decision-making. Monte Carlo Tree Search for strategic planning, ReAct agents for tool-augmented reasoning loops, multi-armed bandits for exploration/exploitation, minimax with alpha-beta pruning for adversarial search, and memory-augmented networks for persistent agent memory.

See 04-agents/README.md for the full algorithm list, timing data, and roadmap.

Script contracts

docs/catalog.json is generated by scripts/generate_catalog.py from explicit per-script records; it is never hand-edited. For the 54 scripts it currently lists, it records:

• Teaching kind — assigned from each script's default program, checked in this order:
  • comparison (17 scripts) trains two or more alternative arms for a common task and reports arm-versus-arm quality or training-cost results, e.g. micrornn (RNN vs GRU), microbatchnorm, microoptimizer, microcheckpoint. Many of these also run inference; comparison is not a no-inference label.
  • train_infer (29) trains one model and then uses it for inference, generation, decisions or evaluation. This includes one trained model run through alternative decoders or serving methods (microbeam, microkv, microquant); a draft or reference model trained only to support decoding, and the stages of one training pipeline (microlora, microdpo), are not separate arms.
  • forward_pass (4: microattention, microflash, micropaged, microrope) runs untrained forward computations.
  • algorithm_demo (4: microbm25, microvectorsearch, microturboquant, micromcts) runs non-learning algorithms.
• Data source — names_download (23 scripts download Karpathy's makemore names.txt on first run) or in_script (31 scripts generate or embed their data).
• Paper slug — the no-magic-papers card each script links to. Script slugs (file basenames such as microgpt) and paper slugs (card names such as gpt-1) are separate namespaces; every link is written out in the generator and never derived from a name.
• Adaptation note — where a script differs from its linked card. For example, microgpt links to the GPT-1 card while its source follows a GPT-2-style architecture with RMSNorm, ReLU and no biases and has no task fine-tuning stage; microvectorsearch implements LSH, not the HNSW index of its card. These notes describe the code as it is; they do not claim paper-scale results, and a script without a note is not thereby claimed to replicate its card exactly.

python scripts/generate_catalog.py --check exits non-zero without writing when the committed catalog is not byte-identical to the generated output. It is a local maintainer command; pull-request CI does not run it today. After changes reach main, the existing Update Catalog workflow (.github/workflows/catalog.yml) regenerates docs/catalog.json and commits any difference; that keeps main current but does not check pull requests.

How to Use This Repo

# Clone the repository
git clone https://github.com/no-magic-ai/no-magic.git
cd no-magic

# Pick any script and run it
python 01-foundations/microgpt.py

That's it. No virtual environments, no dependency installation, no configuration. The 23 scripts whose catalog data_source is names_download fetch makemore's names.txt with urllib on first run, so that first run needs network access; the file is cached as names.txt in the directory you run from and reused from there. The other 30 scripts generate or embed their data and need no network.

Minimum Requirements

• Python 3.10+
• 8 GB RAM
• Any modern CPU (2019-era or newer)

Quick Start Path

If you're working through the scripts systematically, this subset builds core concepts incrementally:

microtokenizer.py     → How text becomes numbers
microembedding.py     → How meaning becomes geometry
microgpt.py           → How sequences become predictions
micrornn.py           → How recurrence models sequences
microlstm.py          → How gated memory solves vanishing gradients
microbert.py          → How bidirectional context differs from autoregressive
microconv.py          → How spatial filters extract features
microvit.py           → How transformers see images
microbatchnorm.py     → How normalizing activations stabilizes training
microlora.py          → How fine-tuning works efficiently
microdpo.py           → How preference alignment works
microattention.py     → How attention actually works (all variants)
microrope.py          → How position gets encoded through rotation
microquant.py         → How models get compressed
microflash.py         → How attention gets fast
microssm.py           → How Mamba models bypass attention entirely
microdiscretize.py    → How discretization shapes what SSMs can learn
microcomplexssm.py    → How complex eigenvalues enable rotation (parity)
microroofline.py      → Why more FLOPs can be faster (SISO → MIMO)
microreact.py         → How agents reason with tools

Each tier's README has the full algorithm list with measured run times for that category.

Learning Resources

Challenges

"Predict the behavior" exercises that test your understanding of the algorithms. 27 challenges covering all 4 tiers — attention, GPT, GAN, DPO, optimizers, discretization, complex SSMs, roofline, tokenizer, embedding, RNN, VAE, LoRA, SFT, PPO, knowledge distillation, ROME knowledge editing, MoE, KV-cache, quantization, TurboQuant, SSM, green-list watermarking, self-consistency voting, eval error bars, MCTS, and ReAct. Each challenge presents a code snippet and asks you to reason about the output before running it.

See challenges/README.md for the full challenge set.

Flashcards

Anki-compatible flashcard decks for spaced repetition review. 238 cards across 4 tiers (foundations, alignment, systems, agents), covering key concepts, equations, and design decisions from every script.

# Generate the Anki decks (needs the genanki package)
uv run --no-project --with genanki python resources/flashcards/generate_anki.py

The script writes no-magic-foundations.apkg, no-magic-alignment.apkg, no-magic-systems.apkg and a combined no-magic-complete.apkg (218 cards) next to itself; agents.csv (20 cards) is not exported. Anki renders card fields as HTML, so angle-bracket notation that a browser would read as a tag, such as an inner product <a,b>, is written with Unicode brackets ⟨a, b⟩ or the entities < and >. A < or > followed by a space or digit, as in < 1, is ordinary visible text and is left as written. See resources/flashcards/ for the raw card data and generation script.

Learning Path

Structured tracks for different goals — 7 learning tracks ranging from weekend sprints to a full curriculum. Each track orders scripts by conceptual dependency and includes time estimates, prerequisites, and milestone markers. In the Transformer, Alignment and Modern Inference tracks every step also gives its exact command, data needs, source/paper/preview links, a predict-before-run question with a checked answer and the limits of what the program shows.

See LEARNING_PATH.md for the full guide.

Offline Book (EPUB)

All 54 scripts compiled into a single EPUB with table of contents, thesis excerpts, tradeoff sections, and full annotated source. Readable on any e-reader, tablet, or phone.

# Requires pandoc: brew install pandoc (macOS) or apt install pandoc
bash scripts/generate-epub.sh
# Output: build/no-magic.epub

A pre-built copy is included in every release.

Translations

Comment translations for 6 languages: Spanish, Portuguese, Chinese, Japanese, Korean, and Hindi. The code stays in English — only comments, docstrings, section headers, and print statements are translated.

See TRANSLATIONS.md for full status and contributor guide.

Want to help translate? See the translation guide.

Dependency Graph

How the algorithms connect conceptually. Arrows mean "understanding A helps with B" — not code imports (every script is fully self-contained).

graph LR
  %% --- Style definitions ---
  classDef foundations fill:#4a90d9,stroke:#2c5f8a,color:#fff
  classDef alignment fill:#e8834a,stroke:#b35f2e,color:#fff
  classDef systems fill:#5bb55b,stroke:#3a823a,color:#fff
  classDef agents fill:#9b59b6,stroke:#7d3c98,color:#fff

  %% === 01-FOUNDATIONS ===
  subgraph F["01 — Foundations"]
    TOK["Tokenizer"]
    EMB["Embedding"]
    OPT["Optimizer"]
    RNN["RNN / GRU"]
    CONV["Conv Net"]
    GPT["GPT"]
    BERT["BERT"]
    RAG["RAG"]
    DIFF["Diffusion"]
    VAE["VAE"]
    GAN["GAN"]
    ERR["Error Bars"]
  end

  %% === 02-ALIGNMENT ===
  subgraph A["02 — Alignment"]
    BN["BatchNorm"]
    DROP["Dropout"]
    SFT["SFT"]
    LORA["LoRA"]
    QLORA["QLoRA"]
    DPO["DPO"]
    REINF["REINFORCE"]
    PPO["PPO"]
    GRPO["GRPO"]
    MOE["MoE"]
    KD["Distillation"]
    ROME["ROME Knowledge Editing"]
  end

  %% === 04-AGENTS ===
  subgraph AG["04 — Agents"]
    MCTS["MCTS"]
    REACT["ReAct"]
  end

  %% === 03-SYSTEMS ===
  subgraph S["03 — Systems"]
    ATTN["Attention"]
    FLASH["Flash Attn"]
    ROPE["RoPE"]
    KV["KV-Cache"]
    PAGED["PagedAttn"]
    QUANT["Quantization"]
    BEAM["Beam Search"]
    CKPT["Checkpointing"]
    PAR["Parallelism"]
    SSM["SSM / Mamba"]
    CSSM["Complex SSM"]
    DISC["Discretize"]
    ROOF["Roofline"]
    TURBO["TurboQuant"]
    WM["Watermarking"]
    COT["Self-Consistency"]
  end

  %% --- Foundation internals ---
  TOK --> GPT
  EMB --> RAG
  RNN --> GPT
  OPT --> GPT
  GPT --> BERT
  DIFF -.-> VAE
  DIFF -.-> GAN
  OPT -.-> ERR
  ERR -.-> COT

  %% --- Foundations → Alignment ---
  GPT --> SFT
  SFT --> DPO
  SFT --> PPO
  SFT -.-> LORA
  GPT --> LORA
  GPT --> DPO
  GPT --> PPO
  GPT --> MOE
  GPT --> GRPO
  LORA --> QLORA
  REINF --> PPO
  REINF --> GRPO
  OPT --> KD
  SFT -.-> KD
  KD -.-> QUANT
  GPT --> ROME
  SFT -.-> ROME
  LORA -.-> ROME
  OPT --> BN
  OPT --> DROP

  %% --- Foundations → Systems ---
  GPT --> ATTN
  GPT --> KV
  GPT --> QUANT
  GPT --> BEAM
  GPT --> SSM
  RNN --> SSM
  ATTN --> FLASH
  ATTN --> ROPE
  KV --> PAGED

  %% --- SSM family connections ---
  SSM --> CSSM
  SSM --> DISC
  SSM --> ROOF
  ROPE -.-> CSSM
  FLASH -.-> ROOF

  %% --- Cross-tier into QLoRA ---
  QUANT --> QLORA

  %% --- Quantization family ---
  QUANT --> TURBO
  EMB -.-> TURBO

  %% --- Decoding-time watermarking ---
  GPT --> WM
  BEAM -.-> WM

  %% --- Reasoning traces and self-consistency decoding ---
  GPT --> COT
  BEAM -.-> COT
  SFT -.-> COT

  %% --- Foundations / Alignment → Agents ---
  REINF --> REACT

  %% --- Apply styles ---
  class TOK,EMB,OPT,RNN,CONV,GPT,BERT,RAG,DIFF,VAE,GAN,ERR foundations
  class BN,DROP,SFT,LORA,QLORA,DPO,REINF,PPO,GRPO,MOE,KD,ROME alignment
  class ATTN,FLASH,ROPE,KV,PAGED,QUANT,BEAM,CKPT,PAR,SSM,CSSM,DISC,ROOF,TURBO,WM,COT systems
  class MCTS,REACT agents

Legend: Foundations · Alignment · Systems · Agents — Solid arrows = strong prerequisite, dashed arrows = conceptual comparison.

Related Projects

• micrograd — Karpathy's autograd engine. The Value class in microgpt.py descends from this.
• makemore — Character-level language modeling. micrornn.py covers similar ground in a single comparative file.

Inspiration & Attribution

This project is directly inspired by Andrej Karpathy's extraordinary work on minimal implementations — particularly micrograd, makemore, and the microgpt.py script that demonstrated the entire GPT algorithm in a single dependency-free Python file.

Karpathy proved that there's enormous demand for "the algorithm, naked." no-magic extends that philosophy across the full landscape of modern AI/ML.

How This Was Built

I built this project with help from multiple AI coding agents. I chose the algorithms, designed the structure and learning paths, set the constraints, and guided the implementation. The agents helped with both the code and the supporting materials, from visualizations and exercises to flashcards, EPUB generation, and translations. I reviewed the results.

This is how I build in 2026. I'd rather be upfront about it.

Star History

Contributing

Contributions are welcome, but the constraints are non-negotiable. See CONTRIBUTING.md for the full guidelines. The short version:

• One file. Zero dependencies. Trains and infers, unless the script's recorded teaching kind is a documented exception.
• If your PR adds a requirements.txt, it will be closed.
• Quality over quantity. Each script should be the best possible minimal implementation of its algorithm.

License

MIT — use these however you want. Learn from them, teach with them, build on them.

---

The constraint is the product. Everything else is just efficiency.

v3.0.0 — April 2026