Large MoE models, powered by your SSD

Website  ·  Blog  ·  Slack  ·  Apache-2.0

macOS / Metal  ·  Linux / CUDA  ·  Windows (in progress)

Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It currently supports DeepSeek-V4.1-Flash-1017, Muse-Spark-Open and Qwen3.6-35B-A3B, with more models coming.

Models

These are the tested GGUF files, community conversions of the named model releases. DeepSeek's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See weight-format details.

Model	Parameter count	GGUF reference
DeepSeek-V4.1-Flash-1017	763B total / 8–16B active	mrfakename · Q8_0
Muse-Spark-Open	816B total / 42B active	Unsloth · UD-Q4_K_XL
Qwen3.6-35B-A3B	35B total / 3B active	Unsloth · UD-Q4_K_XL

Performance benchmarks

4K prompt tier · 256 output tokens · decode measured over tokens 129–256. All rates are in token/s; higher is faster. The multiple next to each InferSSD rate is its speedup over llama.cpp on the same hardware. On the MacBook Pro, Colibri runs the official Safetensors on CPU; llama.cpp and InferSSD run the GGUF with Metal. See measurement details.

Test hardware

Hardware	CPU (cores)	GPU	Memory	SSD
MacBook Pro	Apple M5 Max (18)	40-core	48 GB unified	2 TB internal
RTX 5060 Ti	Intel Core 3 100U (6)	RTX 5060 Ti	8 GB + 16 GB VRAM	2 TB PCIe 4 ×4
RTX 5090	Intel Core Ultra 5 230F (10)	RTX 5090	32 GB + 32 GB VRAM	2 TB PCIe 5 ×4
DGX Spark	NVIDIA GB10 (20)	Blackwell	128 GB unified	4 TB PCIe 5 ×4

Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See hardware details for SSD models, CPU core layouts, and configuration notes.

DeepSeek-V4.1-Flash-1017

Hardware	Prefill (token/s)	Decode (token/s)
llama.cpp	Colibri	InferSSD	llama.cpp	Colibri	InferSSD
MacBook Pro	5.929	0.708	28.766 (4.85×)	0.873	0.437	2.874 (3.29×)
RTX 5060 Ti	6.950	7.020	75.100 (10.81×)	0.242	0.900	2.550 (10.54×)
RTX 5090	8.060	39.210	164.500 (20.41×)	0.867	1.878	7.270 (8.39×)
DGX Spark	2.730	0.826	127.500 (46.70×)	1.608	0.356	7.260 (4.51×)

Muse-Spark-Open

Hardware	Prefill (token/s)	Decode (token/s)
llama.cpp	Colibri	InferSSD	llama.cpp	Colibri	InferSSD
MacBook Pro	3.140	N/S	14.503 (4.62×)	0.153	N/S	1.097 (7.17×)
RTX 5060 Ti	3.410	N/S	52.100 (15.28×)	0.059	N/S	0.219 (3.71×)
RTX 5090	4.000	N/S	106.600 (26.65×)	0.155	N/S	1.520 (9.81×)
DGX Spark	1.740	N/S	83.900 (48.22×)	0.120	N/S	2.140 (17.83×)

N/S: Muse is not currently supported by Colibri.

Get started

Install on Apple Silicon macOS or on Linux with an NVIDIA GPU:

curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

InferSSD currently installs from source, so the engine is compiled for your machine's CPU and GPU; prebuilt packages are not available yet. Building takes about 2 to 12 minutes, depending on the machine.

The installer works in a private environment and uses no sudo. It needs Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs the CUDA toolkit, liburing headers, and systemd with cgroup v2. Follow the printed PATH instruction if it shows one.

Then start a model:

# Chat in your terminal
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

# Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

For Muse, use unsloth/Muse-Spark-Open-816B-A42B-GGUF; for Qwen3.6, unsloth/Qwen3.6-35B-A3B-GGUF. With serve, open http://127.0.0.1:8080 in your browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

The first run downloads the model files (about 470 GiB for DeepSeek or Muse, 21 GiB for Qwen3.6) to ~/.local/share/inferssd/models and measures your machine. Use --model-dir /path/on/ssd to choose the drive, or -m /path/to/first-shard.gguf for files you already have. There is no memory setting to choose: InferSSD uses the memory available. Stop with Ctrl+C, or /exit in chat.

See the quick start for upgrades and uninstalling.

Further reading

• How it works
• Benchmark methodology

Contributing

Contributions are welcome, including hardware results and improvements to expert I/O, caching, and batching. See the contributing guide.

Acknowledgments and license

Built on llama.cpp and ggml, with thanks to their contributors and the model-adaptation community.

InferSSD is licensed under Apache-2.0; bundled llama.cpp/ggml and other third-party components keep their original licenses and notices; see NOTICE. Model weights remain subject to their publishers' licenses.