AkiYangDev/mini-deepinfer
AkiYangDev/mini-deepinfer has 30 GitHub stars and 1 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 21:35 UTC. Last push 2026-10-25.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Learn DeepInfer through a small, runnable inference engine. Follow the code on CPU, then run real Qwen3.5 models on Ascend 910C.
Machine LearningREADME.en.md
OriginalLearn how DeepInfer runs a request.
A small teaching implementation for Ascend 910C, using real Qwen3.5 weights.
简体中文 · English
Quick start · Ascend 910C · Speculative decoding · Fourteen lessons · DeepInfer source map

mini-DeepInfer is a teaching inference engine centered on DeepInfer's execution design. A compact Python and PyTorch implementation connects request scheduling, model execution, sampling and output commits, so you can follow how they work together for one request.
If you can already deploy a model and want to read DeepInfer's source, start here. The fourteen lessons build from a minimal request loop to batching, chunked prefill and cross-round execution. They focus on state after computation is issued: which data the device still needs, when the CPU commits results, and when a cancelled request's cache can be released.
The DeepInfer source map connects upstream entry points to the teaching code and explains the simplifications. After scheduling and in-flight state, follow a rejected proposal in the dual-model walkthrough to see how output commits and cache recovery prepare the next round.
Ascend 910C is the hardware platform; Qwen3.5 models are used to check these mechanisms. Start on CPU with scheduling examples and random tiny models. With the device and weights ready, run real text generation or let 0.8B propose tokens for 2B to verify.
Quick start
Start by watching the engine process requests. This example needs only Python 3.10+, with no dependencies or model weights to install. Run in a terminal:
git clone https://github.com/AkiYangDev/mini-deepinfer.git
cd mini-deepinfer
python -m minideepinfer
The example replaces the model with simple arithmetic: add one to the last input number. It prints the inputs, outputs and slot assignments for each step, ending with:
req-0: [12, 13, 14] (length)
req-1: [21, 22] (eos)
Free slots: 8/8
The first request stops after producing three numbers. The second stops at its configured end marker, 22. All eight slots are returned when both requests finish. This lets you follow submission, execution, output handling and cleanup before loading a language model.
Next, open demo.py and follow engine.submit() into Engine.step(). Lesson one explains the output step by step in Chinese. Then use the DeepInfer source map to connect the teaching implementation to the upstream design.
Where to go next
| What you want to try | Start here |
|---|---|
| Explore scheduling and cache ownership on CPU | Setup → A/B/C walkthrough |
| Run your own prompt on Ascend 910C | Environment and weights → Text generation |
| Understand recovery after a wrong proposal | Round walkthrough → Probability sampling |
The lesson index lists the full sequence, commands and code locations in Chinese. English setup and the A/B/C walkthrough are also available.
Try batching: new requests arrive while a long prompt is still running
After the minimal loop, install PyTorch using setup, then run a randomly initialized tiny Qwen3.5. No checkpoint download is needed:
python -m minideepinfer.pipeline_demo
A's long prompt is processed in chunks, B finishes first, and C reuses B's released cache slots. The A/B/C walkthrough connects each round to the scheduling and cache changes. The full terminal output is also saved. The model is untrained; the example is about execution.
Follow a rejected proposal and cache recovery on CPU
python -m minideepinfer.model_draft_demo --cpu-example rejection --trace-rounds --width 3 --output outputs/draft-rejection.json
The draft moves from six computed inputs back to four valid ones, then consumes the correction to reach five. The round walkthrough follows candidates, actual model inputs and committed output. No checkpoint or NPU is needed. Terminal labels are in Chinese; JSON fields are in English.
Generate text on Ascend 910C
Real Qwen3.5-2B checkpoints have passed single-request, batch and cross-round generation checks. Prepare matching PyTorch, torch_npu and CANN versions and local weights using setup, then run from the repository directory. Multiline commands use Bash:
python -m minideepinfer.generate \
--model weights/Qwen3.5-2B --device npu:0 --dtype float32 \
--prompt "用两句话解释 KV Cache 的作用。" \
--prompt "What is 1 plus 1? Answer in one short sentence." \
--max-new-tokens 128 --cache-capacity 512 \
--max-num-seqs 2 --max-batched-tokens 16 --chunked-prefill
The English answer from the real run:
1 plus 1 equals 2.
[eos; 9 output tokens; free KV slots 512/512]
The complete bilingual output is preserved. Add --pipeline for cross-round execution and --stagger-prompts --trace-rounds to follow arrivals and cache release. See the text frontend.
Speculative decoding and sampling
Let Qwen3.5-0.8B propose and 2B verify. Accept a group and deliver several tokens together; reject it, discard temporary results, restore both caches and continue from the valid prefix. KV, convolution windows and recurrent state must all agree before the next forward pass.
Try your own prompt, watch the output stream, or request cancellation during generation. Complete commands are in the text guide. To understand acceptance under random sampling, start with the small-vocabulary experiment, then examine probability and state-recovery checks on Ascend 910C. This reference path computes the model in NPU FP32 and probabilities in CPU FP64.
Stream output and cancel at a commit boundary
With local Qwen3.5-0.8B and 2B checkpoints:
python -m minideepinfer.generate \
--model weights/Qwen3.5-2B --draft-model weights/Qwen3.5-0.8B \
--speculative --device npu:0 --dtype float32 \
--prompt "The capital of France is" --raw \
--max-new-tokens 10 --cache-capacity 128 --draft-width 3 \
--stream --cancel-after-tokens 2 --check-greedy \
--output outputs/stream-cancel.json
The saved run stopped after delivering a whole group, with four committed tokens and all 14 slots returned in each pool. Cancellation does not truncate an accepted group. The greedy check covers only the delivered prefix, not the remaining output or EOS. Random sampling cannot use --check-greedy.

Read the code alongside DeepInfer
Start with a question, then connect it to the corresponding responsibility in DeepInfer: the engine organizes a step, the Runner prepares inputs and sampling, in-flight state tracks issued work, and result handling decides which outputs and progress to commit. The source map records these connections; the table below gives entry points into the teaching code.
| Mechanism | Code and guide |
|---|---|
| One request: prefill, decode and output commits | Engine.step() · lesson one |
| Batching, token budgets and chunked prefill | BatchEngine · lesson three |
| KV, convolution windows and recurrent state | HybridCache · Qwen3.5 |
| Continuing before the CPU reads the previous token | StoppingDecodePipeline · cross-round chunks |
| Recovering both caches after rejection | ModelDraft · round walkthrough |
| Acceptance and correction sampling | verify_sampled() · sampling experiments |
The fourteen-lesson index provides the full sequence. The DeepInfer source map records corresponding upstream code, retained mechanisms and teaching simplifications.
Already running on Ascend 910C
| Saved record | What it checks |
|---|---|
| Qwen3.5-2B on the NPU | Ascend 910C FP32 model execution, single requests and batching |
| Cross-round text | Matching sync, pipeline and staggered-arrival outputs; 43 full-logit comparisons pass |
| Speculative sampling | 28 probability and six cold-state comparisons pass, including actual draft recovery after rejection |
| Streamed cancellation | Four delivered tokens match an independent greedy prefix; both pools release all slots |
The records preserve inputs, configuration, raw output and source hashes for verification at the corresponding commit. Performance experiments cover sync versus pipeline, first commit and dual-model costs, including short workloads that got slower. Accepting more tokens or making fewer forward calls does not always shorten a request; recovery, copies and waits also take time.
Scope and sources
The commands on this page use main. For a fixed learning version, use v0.5.0, which the setup guide follows. The single-model entry point on main belongs to v0.6 development and supports sampling, streaming and cancellation with one Qwen3.5 checkpoint.
The main learning path uses one device, FP32, eager execution and short contexts. Greedy, temperature and top-k/top-p sampling, streaming and cancellation are implemented. See BF16 notes for reference checks and chunk-path differences. The earlier DeepSeek-R1-Distill-Qwen-1.5B example remains in regression checks. DeepSeek-V4.1 architecture, MLA/MoE, MTP, bounded replay, graphs and distributed execution are not yet implemented; see the roadmap.
Engine design primarily follows DeepInfer. mini-sglang informs the compact teaching format; Ascend SGLang informs NPU API work. References and licenses record the sources.
If the project helps you learn, please give it a Star. If a command fails or an explanation leaves you stuck, open an issue. To add an example or improve a lesson, start with the contribution guide.
This independent community project has no official affiliation with DeepSeek, SGLang or Huawei. Original code and documentation use Apache-2.0; models and third-party code retain their licenses. Character artwork is covered separately in the artwork notes.
