FastLLM

中文 · Quick start · --auto tuning · Documentation · Benchmarks · Changelog

FastLLM is an inference engine for running and serving language models locally. Its C++ runtime does not depend on PyTorch. It supports dense and MoE models, GPU/CPU/NUMA/disk hybrid inference, and multi-GPU deployment.

• Automatic deployment: --auto plans devices, memory, expert caches, and context capacity, with optional overrides.
• Browser launcher: download models, save profiles, inspect logs, and use chat or agents in Studio.
• Compatible APIs: OpenAI Chat Completions, Responses, and Anthropic Messages.
• Multiple weight formats: Safetensors, FastLLM, AWQ, and supported GGUF models. See the model guides for coverage.

Quick start

Installation

Linux / Windows with NVIDIA GPUs:

python -m pip install -U ftllm

python -m pip install https://www.modelscope.cn/models/huangyuyang/fastllmdepend-windows/resolve/master/ftllmdepend-0.0.0.2-py3-none-win_amd64.whl

For AMD GPUs, see ROCm installation (Chinese). For CPU-only and other platforms, see building from source. Installation help is in the FAQ (Chinese).

Browser launch

For multiple independent replicas on one Linux host, ftllm deploy adds background process supervision and streaming request routing while reusing native server arguments. Existing ftllm serve/server behavior is unchanged. See deployment configuration.

# API server, listening on 0.0.0.0:8080 by default
ftllm server Qwen/Qwen3-0.6B

# Command-line chat
ftllm run Qwen/Qwen3-0.6B

# WebUI, connected to the API server above; listens on 127.0.0.1:1616 by default
ftllm webui --api_base http://127.0.0.1:8080/v1

Run the browser deployment launcher directly:

ftllm

Select a model → choose Automatic configuration → save and start. Expand Resource limits or Model parameters only when needed. Once ready, open Studio to chat. See the launcher guide.

Command-line launch

Replace the path with a local model directory or supported GGUF file. Hugging Face repository IDs also work:

ftllm server /path/to/model --auto

--auto currently supports selected models on Linux with NVIDIA CUDA. Other platforms and models use manual deployment. The default local service URL is http://127.0.0.1:8080.

Append only the options you want to change. For example, a 32K session window and at most 2 concurrent requests:

ftllm server /path/to/model --auto --max_context_length 32768 --max_batch 2

See the --auto tuning guide for memory budgets, expert caches, chunks, speculative decoding, and fast prefill. Use ftllm run for CLI chat, ftllm tui for the terminal wizard, and ftllm bench for benchmarks.

Call the API

You can set a service name and API key at startup:

ftllm server /path/to/model --auto --model_name local-model --api_key local-key

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer local-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Hello"}],"stream":false}'

For standalone chat, run ftllm webui --api_base http://127.0.0.1:8080/v1 --api_key local-key. Reasoning, tool calling, and multimodal features depend on the model.

Model deployment guides

Other models are listed in the support list (Chinese). For agent-assisted deployment, provide the model and option reference (Chinese). Measured speed depends on hardware, workload, and quantization; see the benchmarks.

Common options

Start with --auto options and effects. For fixed devices or model-specific settings, use the manual option reference. ftllm server --help is authoritative for the installed version.

Device settings · Decoding, templates, and tools · Download and export

Build from source

See the source-build instructions for C++ / CUDA / CPU builds, and the FAQ (Chinese) for platform troubleshooting.

Documentation

• All documentation · Launcher and Studio · Agent integrations (Chinese)
• Hybrid inference (Chinese) · Expert caches (Chinese) · Context extension (Chinese)
• Development and validation (Chinese) · Custom models (Chinese) · Custom operators (Chinese)
• Stable changelog · Nightly (Chinese)

Community and contributing

Report problems through Issues and contribute through Pull Requests. Deployment discussion QQ group: 831641348.

User and developer WeChat groups

User discussion (usage and deployment):

Developer community (contributions and development):

FastLLM is licensed under the Apache License 2.0.

Acknowledgements and references