We want this Chinese adaptation of CS336 to be more than a translated version of Stanford's original course. Our goal is to build an “LLM alchemy workshop” tailored to Chinese-speaking learners—a place where you forge your understanding, refine your code, master the training process, and ultimately create a large language model of your own.

📋 Prerequisites

• Python programming: Proficiency in Python and solid software engineering skills
• Deep learning fundamentals: Familiarity with PyTorch and the basic principles of neural networks
• Mathematics: Linear algebra, probability and statistics, and calculus
• Machine learning: A solid grasp of the fundamentals of machine learning and deep learning
• GPU programming (optional): Familiarity with basic CUDA concepts is helpful but not required; this project also includes introductory material

📚 Course Vision

• A balance of rigorous theory and hands-on practice: We retain the technical depth of the original course while reorganizing the material around the learning habits of Chinese-speaking students. We also cover essential mathematical and deep learning prerequisites to provide a smoother learning curve.
• A progressive knowledge system: The large engineering challenge of building an LLM is broken down into approachable, understandable modules. By the end of the course, you will have a complete mental map of LLM construction.
• Code-driven learning: The core philosophy of the course is to “think through code.” In addition to implementations for every assignment, we share the reasoning behind the code.
• Localization for the Chinese ecosystem: We account for local network conditions, available computing resources, and the domestic open-source ecosystem, with practical solutions and examples involving excellent models such as Qwen and DeepSeek.

🎯 What You Will Gain

What can you expect after completing the course?

• A solid technical foundation: You will be able to build your own LLM and understand every core component.
• Valuable engineering experience: You will practice the full workflow—from data processing and model training to deployment and optimization.
• Stronger industry competitiveness: You will develop the core skills required for large-model research and engineering roles.
• A clear research perspective: You will gain a systematic understanding of the LLM field and a strong foundation for further research.

📖 Course Outline

Chapter	Key Topics	Assignment	Status
Foreword	Project background, learning path, and prerequisites	-	✅
Chapter 1: Tools	W&B experiment tracking, hyperparameter search, and visualization dashboards	-	📝
Chapter 2: Tokenizers	BPE, Unicode normalization, and implementing tokenizer training from scratch	Assignment 1	✅
Chapter 3: PyTorch and Resource Accounting	Mixed-precision training, gradient accumulation, FLOPs, and memory estimation	-	✅
Chapter 4: Language Model Architecture and Training Details	RoPE, RMSNorm, SwiGLU, AdamW, Pre-Norm vs. Post-Norm, and learning-rate schedules	Assignment 1	✅
Chapter 5: Mixture of Experts	Top-K routing, load balancing, auxiliary loss, Expert Parallelism, and DeepSeekMoE	-	✅
Chapter 6: GPUs and Related Optimizations	Memory bandwidth, arithmetic intensity, FlashAttention, kernel fusion, mixed precision, BF16, and TF32	Assignment 2	✅
Chapter 7: High-Performance GPU Programming	CUDA programming, Tensor Cores, shared memory, and an introduction to Triton	Assignment 2	✅
Chapter 8: Distributed Training	Data/model/pipeline parallelism, ZeRO-1/2/3, FSDP, gradient accumulation, and All-Reduce	Assignment 2	✅
Chapter 9: Scaling Laws	Chinchilla laws, compute-optimal configurations, scaling experiment design, and extrapolation	Assignment 3	✅
Chapter 10: Inference	KV cache, speculative decoding, quantization (GPTQ/AWQ), PagedAttention, and continuous batching	Assignment 6	✅
Chapter 11: Data Engineering	Data-quality filtering, MinHash deduplication, PII removal, data mixtures, and data curricula	Assignment 4	✅
Chapter 12: Evaluation and Benchmarking	MMLU, HumanEval, HELM, CEval, AlpacaEval, and Arena rankings	Assignment 6	✅
Chapter 13: The Basic LLM Training Pipeline	Pre-training, SFT, DPO, RLHF with PPO, and the alignment tax	Assignment 5	✅
Chapter 14: Reinforcement Learning with Verifiable Rewards	GRPO, rule-based verifiers, outcome/process rewards, RLVR, and R1-style training	Assignment 5	✅
Chapter 15: Multimodal Models	From CLIP to omni models: CLIP/SigLIP, LLaVA, Qwen-VL, and Chameleon	-	✅
Chapter 16: Extended Topics	1. What Is LLM Reasoning? (Youzhen Li) 2. The Future of LLMs—LeCun (Youzhen Li, Jiang Yinhe, Shengkang Li, and Hu Xu)	-	🔄

Status: ✅ Complete · 🔄 Updating · 📝 Needs improvement · 🚧 In preparation · ⏸️ Paused

📝 Assignment Overview

Assignment	Core Tasks	Status
Assignment 1: Build an LLM from Scratch	Implement a tokenizer, model architecture, and optimizer, then train a minimal language model	✅
Assignment 2: Systems Optimization	Performance analysis and benchmarking; implement FlashAttention-2 in Triton; build distributed training code	✅
Assignment 3: Scaling Laws	Understand Transformer components and fit scaling laws to predict model-scaling outcomes	✅
Assignment 4: Data Processing	Convert raw Common Crawl data into a pre-training dataset, including filtering and deduplication	✅
Assignment 5: Model Alignment	Apply SFT and reinforcement learning methods such as GRPO to train models for mathematical reasoning	✅
Assignment 6: Model Evaluation	Use lm-evaluation-harness and EvalScope for multidimensional evaluation across language understanding, commonsense reasoning, coding, and mathematics	✅

🚀 Quick Start

# Clone the repository
git clone https://github.com/datawhalechina/diy-llm.git
cd diy-llm
# Install the dependencies required by the assignment you are working on

Learning Path

1️⃣ Study the theory → Read the English material in docs/en/ in chapter order (the Chinese version is in docs/zh/)
2️⃣ Practice → Complete the six assignments in coursework/
3️⃣ Deepen your understanding → Read the implementations and study the design of each component

Project Structure

diy-llm/
├── docs/                           # Online course material
│   ├── zh/                         # Chinese documentation (default)
│   │   ├── 前言.md
│   │   ├── chapter1/               # Tools
│   │   ├── chapter2/               # Tokenizers
│   │   ├── chapter3/               # PyTorch and resource accounting
│   │   ├── chapter4/               # Architecture and training details
│   │   ├── chapter5/               # Mixture of Experts
│   │   ├── chapter6/               # GPUs and optimization
│   │   ├── chapter7/               # High-performance GPU programming
│   │   ├── chapter8/               # Distributed training
│   │   ├── chapter9/               # Scaling laws
│   │   ├── chapter10/              # Inference
│   │   ├── chapter11/              # Data engineering
│   │   ├── chapter12/              # Evaluation and benchmarking
│   │   ├── chapter13/              # LLM training pipeline
│   │   ├── chapter14/              # Reinforcement learning with verifiable rewards
│   │   ├── chapter15/              # Multimodal models
│   │   └── chapter16/              # Extended topics
│   ├── en/                         # English documentation
│   │   └── ...
│   └── .vitepress/                 # VitePress configuration
├── coursework/                     # Hands-on assignments
│   ├── assignment1-basics/         # Assignment 1: Build an LLM from scratch
│   ├── assignment2-systems/        # Assignment 2: Systems optimization
│   ├── assignment3-scaling/        # Assignment 3: Scaling laws
│   ├── assignment4-data/           # Assignment 4: Pre-training data processing
│   ├── assignment5-alignment/      # Assignment 5: Alignment
│   └── assignment6-evaluation/     # Assignment 6: Evaluation
├── README.md                       # Chinese project overview
├── README_EN.md                    # English project overview
└── .gitignore                      # Git ignore rules

PDF Download

A PDF edition is available for offline reading and printing. To discourage unauthorized resale with third-party watermarks, the PDF contains a subtle Datawhale open-source watermark that does not affect readability. Thank you for your understanding.

📥 Diy-LLM Course Notes PDF: https://github.com/datawhalechina/diy-llm/releases/latest/

🔗 Related Links

• Repository: https://github.com/datawhalechina/diy-llm
• Read online: https://datawhalechina.github.io/diy-llm/
• Original course website: Stanford CS336 (Spring 2026)
• Original course repository: https://github.com/stanford-cs336/lectures/tree/main

❓ Frequently Asked Questions

Q: Can I study the course without a GPU?

Yes. You can study all theoretical material and debug parts of the assignments on a CPU, although full training runs require a GPU. A cloud GPU service is recommended if you do not have local access to one.

Q: How does this project differ from the original CS336 course?

While preserving the technical depth of the original course, we have localized it for Chinese-speaking learners through Chinese explanations, assignment implementations, more detailed references, and examples featuring models developed in China.

💬 Reader Community

Join the Diy-LLM reader groups to study, exchange ideas, and help one another. If you would like to join, please add one of the coordinators below on WeChat and include “diy-llm” in your verification message.

Coordinator	WeChat ID
Hu Xu	xuhu96736
Youzhen Li	zydsx111
Shengkang Li	muzichengminguangli

👥 Contributors

Core Contributors

Hu Xu Project Lead Datawhale Member Shanghai Jiao Tong University Contributions: Chapters 1, 3, 9, 12, 14, and 15; Assignments 5 and 6; full-content review

Shengkang Li Project Lead Datawhale Member Xi'an University of Posts and Telecommunications Contributions: Chapters 4, 6, 7, 8, and 13; Assignments 2 and 4

Youzhen Li Project Lead Datawhale Member Contributions: Chapters 2, 5, 10, 11, and 13; Assignments 1 and 3

Community Contributions

• Thanks to @aimetrics for adding MPS support for MacBook devices to Assignment 1's train.py.
• Thanks to @FuTseYi, a Datawhale member, for supporting the website migration, deployment, and refactoring.
• Thanks to @jiangyinhe for helping prepare Chapter 16's extended topic “The Future of LLMs—LeCun,” and for translating the Foreword and Chapter 1 into English from the Chinese material.
• Thanks to @Excelius-Wang for adding an introduction to the C-EVAL Chinese benchmark and integrating it into Assignment 6.

We appreciate every developer who has contributed to this project, and we welcome community members to get involved!

Contributions of all kinds are welcome. Improvements to the documentation, code optimizations, bug fixes, and new content are all valuable to the project.

How to Contribute

1. Report issues: If you find documentation errors, bugs, or opportunities for improvement, please open an Issue.
2. Submit code: Fork the repository, create a feature branch, commit your changes, and open a Pull Request.
3. Improve the documentation: Help refine existing material, translate content, or add examples.
4. Share your experience: Post your learning notes and practical experience in the discussions area.

Contribution Guidelines

• Keep the code style consistent with the existing project.
• Follow the current documentation format when adding new content.
• Provide a clear description and change summary in each Pull Request.
• For substantial changes, discussion in an Issue is encouraged before implementation.

📝 Changelog

The project is under continuous development. See GitHub Releases or the commit history for the latest updates.

📄 License

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

🙏 Acknowledgements

• Thanks to the Stanford CS336 course team for creating the excellent original course.
• Special thanks to @Sm1les for helping and supporting this project.
• Thanks to every developer who has contributed to the project.
• Thanks to the open-source community for its support and feedback.

⭐ Star History

If this project helps you, please consider giving it a Star ⭐️!

---