GeeeekExplorer/nano-vllm

GeeeekExplorer/nano-vllm has 16,388 GitHub stars and 2,856 forks. GitHot measured +101 in 7 days. Last checked 2026-11-11 09:42 UTC. Last push 2026-04-26.

这些数字如何统计

Nano vLLM

16,388 星标 2,856 forks

可以做什么

🚀 Fast offline inference - Comparable inference speeds to vLLM
📖 Readable codebase - Clean implementation in ~ 1,200 lines of Python code
⚡ Optimization Suite - Prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.

实测星标增长

24小时: 更新延迟
7天: +101
15天: 正在收集历史
1个月: 正在收集历史

各周期内的星标净变化，周期相互重叠，并非每日新增。缺失的测量保持空白。星标代表关注度，不代表软件质量。

README.md

原文

Nano-vLLM

A lightweight vLLM implementation built from scratch.

Key Features

🚀 Fast offline inference - Comparable inference speeds to vLLM
📖 Readable codebase - Clean implementation in ~ 1,200 lines of Python code
⚡ Optimization Suite - Prefix caching, Tensor Parallelism, Torch compilation, CUDA graph, etc.

Installation

pip install git+https://github.com/GeeeekExplorer/nano-vllm.git

Model Download

To download the model weights manually, use the following command:

huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
  --local-dir ~/huggingface/Qwen3-0.6B/ \
  --local-dir-use-symlinks False

Quick Start

See example.py for usage. The API mirrors vLLM's interface with minor differences in the LLM.generate method:

from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]