Skip to content
Projects

AkiYangDev/mini-deepinfer has 30 GitHub stars and 1 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 21:35 UTC. Last push 2026-10-25.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Learn DeepInfer through a small, runnable inference engine. Follow the code on CPU, then run real Qwen3.5 models on Ascend 910C.

Machine Learning

README.en.md

Original

mini-DeepInfer

Learn how DeepInfer runs a request.

A small teaching implementation for Ascend 910C, using real Qwen3.5 weights.

CPU checks Ascend 910C Python 3.10+ Apache 2.0

简体中文 · English

Quick start · Ascend 910C · Speculative decoding · Fourteen lessons · DeepInfer source map

Blue-haired, whale-themed DeepSeek girl illustration

mini-DeepInfer is a teaching inference engine centered on DeepInfer's execution design. A compact Python and PyTorch implementation connects request scheduling, model execution, sampling and output commits, so you can follow how they work together for one request.

If you can already deploy a model and want to read DeepInfer's source, start here. The fourteen lessons build from a minimal request loop to batching, chunked prefill and cross-round execution. They focus on state after computation is issued: which data the device still needs, when the CPU commits results, and when a cancelled request's cache can be released.

The DeepInfer source map connects upstream entry points to the teaching code and explains the simplifications. After scheduling and in-flight state, follow a rejected proposal in the dual-model walkthrough to see how output commits and cache recovery prepare the next round.

Ascend 910C is the hardware platform; Qwen3.5 models are used to check these mechanisms. Start on CPU with scheduling examples and random tiny models. With the device and weights ready, run real text generation or let 0.8B propose tokens for 2B to verify.


Quick start

Start by watching the engine process requests. This example needs only Python 3.10+, with no dependencies or model weights to install. Run in a terminal:

git clone https://github.com/AkiYangDev/mini-deepinfer.git
cd mini-deepinfer
python -m minideepinfer

The example replaces the model with simple arithmetic: add one to the last input number. It prints the inputs, outputs and slot assignments for each step, ending with:

req-0: [12, 13, 14] (length)
req-1: [21, 22] (eos)
Free slots: 8/8

The first request stops after producing three numbers. The second stops at its configured end marker, 22. All eight slots are returned when both requests finish. This lets you follow submission, execution, output handling and cleanup before loading a language model.

Next, open demo.py and follow engine.submit() into Engine.step(). Lesson one explains the output step by step in Chinese. Then use the DeepInfer source map to connect the teaching implementation to the upstream design.

Where to go next

What you want to try Start here
Explore scheduling and cache ownership on CPU Setup → A/B/C walkthrough
Run your own prompt on Ascend 910C Environment and weights → Text generation
Understand recovery after a wrong proposal Round walkthrough → Probability sampling

The lesson index lists the full sequence, commands and code locations in Chinese. English setup and the A/B/C walkthrough are also available.

Try batching: new requests arrive while a long prompt is still running

After the minimal loop, install PyTorch using setup, then run a randomly initialized tiny Qwen3.5. No checkpoint download is needed:

python -m minideepinfer.pipeline_demo

Actual CPU output: B finishes and C reuses its cache slots

A's long prompt is processed in chunks, B finishes first, and C reuses B's released cache slots. The A/B/C walkthrough connects each round to the scheduling and cache changes. The full terminal output is also saved. The model is untrained; the example is about execution.

Follow a rejected proposal and cache recovery on CPU
python -m minideepinfer.model_draft_demo --cpu-example rejection --trace-rounds --width 3 --output outputs/draft-rejection.json

The draft moves from six computed inputs back to four valid ones, then consumes the correction to reach five. The round walkthrough follows candidates, actual model inputs and committed output. No checkpoint or NPU is needed. Terminal labels are in Chinese; JSON fields are in English.

Generate text on Ascend 910C

Real Qwen3.5-2B checkpoints have passed single-request, batch and cross-round generation checks. Prepare matching PyTorch, torch_npu and CANN versions and local weights using setup, then run from the repository directory. Multiline commands use Bash:

python -m minideepinfer.generate \
  --model weights/Qwen3.5-2B --device npu:0 --dtype float32 \
  --prompt "用两句话解释 KV Cache 的作用。" \
  --prompt "What is 1 plus 1? Answer in one short sentence." \
  --max-new-tokens 128 --cache-capacity 512 \
  --max-num-seqs 2 --max-batched-tokens 16 --chunked-prefill

The English answer from the real run:

1 plus 1 equals 2.

[eos; 9 output tokens; free KV slots 512/512]

The complete bilingual output is preserved. Add --pipeline for cross-round execution and --stagger-prompts --trace-rounds to follow arrivals and cache release. See the text frontend.

Speculative decoding and sampling

Let Qwen3.5-0.8B propose and 2B verify. Accept a group and deliver several tokens together; reject it, discard temporary results, restore both caches and continue from the valid prefix. KV, convolution windows and recurrent state must all agree before the next forward pass.

Try your own prompt, watch the output stream, or request cancellation during generation. Complete commands are in the text guide. To understand acceptance under random sampling, start with the small-vocabulary experiment, then examine probability and state-recovery checks on Ascend 910C. This reference path computes the model in NPU FP32 and probabilities in CPU FP64.

Stream output and cancel at a commit boundary

With local Qwen3.5-0.8B and 2B checkpoints:

python -m minideepinfer.generate \
  --model weights/Qwen3.5-2B --draft-model weights/Qwen3.5-0.8B \
  --speculative --device npu:0 --dtype float32 \
  --prompt "The capital of France is" --raw \
  --max-new-tokens 10 --cache-capacity 128 --draft-width 3 \
  --stream --cancel-after-tokens 2 --check-greedy \
  --output outputs/stream-cancel.json

The saved run stopped after delivering a whole group, with four committed tokens and all 14 slots returned in each pool. Cancellation does not truncate an accepted group. The greedy check covers only the delivered prefix, not the remaining output or EOS. Random sampling cannot use --check-greedy.

A small glowing-eyed chibi whale girl Easter egg

Read the code alongside DeepInfer

Start with a question, then connect it to the corresponding responsibility in DeepInfer: the engine organizes a step, the Runner prepares inputs and sampling, in-flight state tracks issued work, and result handling decides which outputs and progress to commit. The source map records these connections; the table below gives entry points into the teaching code.

Mechanism Code and guide
One request: prefill, decode and output commits Engine.step() · lesson one
Batching, token budgets and chunked prefill BatchEngine · lesson three
KV, convolution windows and recurrent state HybridCache · Qwen3.5
Continuing before the CPU reads the previous token StoppingDecodePipeline · cross-round chunks
Recovering both caches after rejection ModelDraft · round walkthrough
Acceptance and correction sampling verify_sampled() · sampling experiments

The fourteen-lesson index provides the full sequence. The DeepInfer source map records corresponding upstream code, retained mechanisms and teaching simplifications.

Already running on Ascend 910C

Saved record What it checks
Qwen3.5-2B on the NPU Ascend 910C FP32 model execution, single requests and batching
Cross-round text Matching sync, pipeline and staggered-arrival outputs; 43 full-logit comparisons pass
Speculative sampling 28 probability and six cold-state comparisons pass, including actual draft recovery after rejection
Streamed cancellation Four delivered tokens match an independent greedy prefix; both pools release all slots

The records preserve inputs, configuration, raw output and source hashes for verification at the corresponding commit. Performance experiments cover sync versus pipeline, first commit and dual-model costs, including short workloads that got slower. Accepting more tokens or making fewer forward calls does not always shorten a request; recovery, copies and waits also take time.

Scope and sources

The commands on this page use main. For a fixed learning version, use v0.5.0, which the setup guide follows. The single-model entry point on main belongs to v0.6 development and supports sampling, streaming and cancellation with one Qwen3.5 checkpoint.

The main learning path uses one device, FP32, eager execution and short contexts. Greedy, temperature and top-k/top-p sampling, streaming and cancellation are implemented. See BF16 notes for reference checks and chunk-path differences. The earlier DeepSeek-R1-Distill-Qwen-1.5B example remains in regression checks. DeepSeek-V4.1 architecture, MLA/MoE, MTP, bounded replay, graphs and distributed execution are not yet implemented; see the roadmap.

Engine design primarily follows DeepInfer. mini-sglang informs the compact teaching format; Ascend SGLang informs NPU API work. References and licenses record the sources.

If the project helps you learn, please give it a Star. If a command fails or an explanation leaves you stuck, open an issue. To add an example or improve a lesson, start with the contribution guide.

This independent community project has no official affiliation with DeepSeek, SGLang or Huawei. Original code and documentation use Apache-2.0; models and third-party code retain their licenses. Character artwork is covered separately in the artwork notes.