Skip to content
Projects

RustingStudio/RustingStrata has 250 GitHub stars and 31 forks. GitHot measured +18 stars in 24 hours. Last checked 2026-11-15 23:35 UTC. Last push 2026-11-15.

How these counts are measured

Measured star growth

in 24h
+18
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

AI

README.md

Original

RustingStrata

A pure-Rust inference engine for the Qwen3.5/3.6 hybrid model family (qwen35moe, qwen35) in GGUF; Qwen3-Next (qwen3next), Qwen3.8-Flash-Next (qwen4exp) and Qwen3 (qwen3, qwen3moe) also run. CUDA kernels are compiled at run time with NVRTC through cudarc (cached as a cubin), and the same kernels also run on Vulkan (--backend vulkan, GLSL compiled to SPIR-V and committed): no C++ build step, no ggml. Experts that do not fit in VRAM run on the CPU at the same time as the GPU work, and the expert placement adapts to the routing it sees.

Status: v1.2.1. Milestones M0-M6 are done (M6: server, chat, calibration, bench); the v1 gate run is in docs/m6-results.md.

Numbers

Reference machine: RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM. Model: Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf (11.7 GiB, does not fit in VRAM). llama.cpp is the 2026-10-28 build 11876, tuned for the same file (-ncmoe 12 -ub 2048 -b 2048, flash attention; with its default ubatch its 4K prefill is ~1550).

CUDA Vulkan llama.cpp
Prefill, 4K prompt 2216 tok/s (2169-2252) 2154 (2140-2162) ~2236 (2183-2310)
Decode after 4K prompt 113.4 tok/s (110.9-114.8) 113.6 66.5
Decode, short prompt (like tg128) 118.2 tok/s (117.0-118.8) 117.3 68.4
Decode with MTP drafts (--mtp 1, greedy), short / after 4K up to 160 / 152 tok/s 158 / 154 (its MTP is slower)
32K: prefill / decode 1800 / 91.9 tok/s 1582 / 90.7
Greedy top-1 agreement with llama.cpp (20 prompts x 32 tokens), either llama.cpp mode 98-99% 98-99% 98.3% (its batch vs token-by-token modes)

Measured with strata bench --gpu --mtp 0 --pp 512,4096,32768 --tg 128 --reps 3 --ctx 32896, mean of 3 rounds (range in brackets); llama.cpp from llama-server timings, 4047-token prompt + 128 tokens, 3 runs. MTP gains depend on the text. Agreement counts a token when it matches llama.cpp run token by token or with the context as one batch; llama.cpp's own two modes agree on only 98.3%. The remaining misses are near-ties where float rounding order picks a different token. The expert placement adapts between runs and the CPU and GPU expert kernels round differently, so greedy output can change at such near-ties from one run to the next. MTP on and off give identical greedy tokens.

With less VRAM, simulated on the 3060 with --vram-margin (not measured on real 8 GB or 6 GB cards): ~8 GB keeps 116 / 110 tok/s decode (short / after 4K) and 2128 tok/s 4K prefill, ~6 GB 105 / 97 and 2056.

Other models, greedy "The capital of France is" with --gpu (CUDA / Vulkan, second run):

Model Types Decode
Qwen3.5-2B (fine-tune) Q4_K_M Q4_K, Q6_K, ... 208 / 208 tok/s
Qwen3-4B-Instruct-2507 Q4_K_M (dense) Q4_K, Q6_K 104 / 110 tok/s
Qwen3.5-9B Q5_K_S Q5_K, Q6_K, Q8_0 51 / 53 tok/s
Qwen3-30B-A3B-Instruct-2507 UD-Q2_K_XL Q2_K to Q6_K 114 / 123 tok/s
Qwen3.8-27B UD-IQ3_XXS (dense, 10.8 GB on GPU) IQ1_S to IQ4_XS, Q2_K to Q8_0 20.6 / 20.5 tok/s
Qwen3.6-35B-A3B UD-IQ4_XS (57% of experts fit in VRAM) IQ3_S, IQ4_XS, Q6_K, Q8_0 94 / 96 tok/s
Qwen3-Next-80B-A3B-Instruct UD-IQ1_S (most experts on the CPU) IQ1_S to IQ4_NL, Q4_K to Q8_0, BF16 92 / 90 tok/s
Qwen3.8-Flash-Next GSQ-RCO IQ2_XS (64 GB in two files, pass the -00001-of-00002 one; 18% of experts fit in VRAM, and with 32 GB RAM the rest are partly read from disk) Q2_0, IQ1_M to IQ4_XS, Q6_K, Q8_0, F16, BF16 22 / 20 tok/s

Supported GGUF types: F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, Q2_0, Q2_K-Q6_K, IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL.

Build

Needs Rust (2024 edition) and either an NVIDIA driver with the CUDA NVRTC library (loaded at run time; CUDA 13 dropped Volta/sm_70, so a V100 needs the 12.x toolkit) or a Vulkan 1.3 driver with shaderFloat64, int8/16/64 and subgroup size 32 or a fixed size 64 (tested on RTX 3060, Radeon VII and V100; Intel Arc lacks shaderFloat64). The CUDA kernels run on Volta (sm_70, m8n8k4 tensor cores), Turing (sm_75) and Ampere or newer (sm_80+, cp.async and int8 tensor cores); Pascal and older use --backend vulkan. --gpu picks CUDA when it loads, else Vulkan; --backend chooses. The first CUDA run compiles the kernels (~15 s) and caches them. AMD GPUs: see docs/AMD.md for known issues and what to send in a report. Every GPU that has been tried is listed in docs/SUPPORTED_GPUS.md.

cargo build --release

Windows: install the Visual Studio C++ Build Tools first (one dependency compiles C code).

Use

Without a terminal: run strata (double-click strata.exe on Windows; release binaries are on the GitHub releases page). It starts the server on http://127.0.0.1:8080 and opens a chat page in the browser. The page keeps chats in the browser, downloads models from Hugging Face (Models and settings; HF_TOKEN for gated repos), switches between the models in the models folder (%LOCALAPPDATA%\rusting_strata\models on Windows, ~/.local/share/rusting_strata/models elsewhere, or serve --models-dir), and shows the context use, speed, totals and CPU, memory and graphics memory use of the server. Other apps that take an OpenAI-compatible address (Open WebUI, Jan, SillyTavern, code editors) can use http://127.0.0.1:8080/v1 too.

From a terminal:

M=path/to/Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf

# Measure the best prefill chunk, short-prompt threshold and MTP drafts on this machine once.
# Later runs read ~/.cache/rusting_strata/strata-<model>.json unless flags override it.
target/release/strata calibrate --model $M

target/release/strata generate --model $M --gpu --prompt "Write a haiku about rivers." -n 128
target/release/strata chat --model $M --gpu                 # --think for thinking mode, --temp 0.7 to sample
target/release/strata bench --model $M --gpu --pp 512,4096 --tg 128
STRATA_API_KEY=secret target/release/strata serve --model $M --gpu --addr 127.0.0.1:8080

serve speaks the OpenAI chat completions API (POST /v1/chat/completions, streamed or not, GET /v1/models). It runs one request at a time; --parallel N (GPU) decodes up to N together (4 requests: 164 tok/s in total on the 3060, 41 each; see docs/HOSTING.md). temperature, top_k, top_p and seed are honored, and decoding is greedy when they are left out. max_tokens (or max_completion_tokens) and stop (up to 16 strings) work as in the OpenAI API; n must be 1. Thinking is off unless a request sets chat_template_kwargs.enable_thinking (Anthropic: thinking: {"type": "enabled"}). When a request repeats the previous request's messages and adds new ones, only the new part is prefilled (usage.prompt_tokens_details.cached_tokens). tools and tool_choice work as in the OpenAI API (calls come back as tool_calls with finish_reason: "tool_calls"). The prompt follows the model's own chat template, rendered like llama.cpp's --jinja. tool_choice: "required" does not force a call. POST /v1/messages speaks the Anthropic Messages API (text, tool_use and tool_result blocks, streamed or not, the key in x-api-key, and POST /v1/messages/count_tokens), so Claude Code can use the server: ANTHROPIC_BASE_URL=http://127.0.0.1:8080 ANTHROPIC_API_KEY=secret ANTHROPIC_MODEL=local ANTHROPIC_SMALL_FAST_MODEL=local claude. OpenCode and other agents use the OpenAI address. Qwen3.6's template leaves the think block out of replies before the last user message, so each new user message changes the text of the earlier turns and the whole conversation is prefilled again. Strata keeps those blocks by default, so a long agent session prefills only what was added; --preserve-thinking false (or chat_template_kwargs.preserve_thinking in a request) follows the template. A request with another history (an agent's side request for a title or a summary) parks the conversation's state in host memory (--cache-ram, 4096 MB by default), and the next request that continues it takes it back instead of prefilling it again. For a server that other people use (customer keys in a file, a usage log for billing, /health, TLS through a reverse proxy, systemd), see docs/HOSTING.md. Each customer key reuses only its own cached conversations; all requests under one key count as one user.

Useful flags: --ctx (KV cache size, default 4096), --mtp [N] (speculative decoding with the model's draft head; greedy tokens stay the same, sampling keeps its distribution), --chunk, --threads, --vram-margin. On a GPU that drives no display, --vram-margin 256 keeps more experts resident (+1-2% prefill); 128 runs out of memory. --kv q8 halves the KV cache (345 instead of 650 MB at 32K) at about 6% slower 32K decode and 14% slower 32K prefill. Without --gpu everything runs on the CPU reference, which is slow and meant for checking. If you reach serve through a LAN name while it is bound to 0.0.0.0, allow that name with --allowed-hosts ai-server.lan (or STRATA_ALLOWED_HOSTS=ai-server.lan).

Several GPUs: --device 0,1 splits the layers over the cards in proportion to their free VRAM, so more experts stay resident. The cards run one after the other, so one request gets more memory, not twice the speed of one card. MTP drafting is off on a split model. AMD cards clock down in the gaps while the other card works; echo high > /sys/class/drm/cardN/device/power_dpm_force_performance_level (as root, per card) took a two-card split from 44 to 57 tok/s in one report.

The experts that do not fit in VRAM are read from the GGUF through the page cache. A file system that keeps file data out of the page cache (ZFS with primarycache=metadata) makes every run read them from disk again; load the model from a file system that caches data, or from a tmpfs copy.

DeepSeek-V4.1-Flash (IQ2_XXS, 249 GB)

chat, generate and serve detect the deepseek41 architecture and run it on the CPU reference forward with the matvecs and a 9 GB expert cache on the CUDA card (no --gpu flag needed). Pass the first shard; the others are found next to it.

M=DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf
strata chat     --model $M [--think]            # multi-turn, KV cache kept between turns
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080 [--api-key K]   # /v1/chat/completions, JSON and SSE
DeepSeek-V4.1-Flash IQ2_XXS (249 GB), RTX 3060 12 GB, 30 GB RAM RustingStrata llama.cpp b12116 (--cpu-moe)
Decode 1.8 tok/s at first, 2-3.6 tok/s now (VRAM expert cache with LRU) 0.3-1.5 tok/s (cold)
Prefill ~2 tok/s on short prompts (0.5 s/token), ~0.5 tok/s at 1400 tokens 0.5-0.9 tok/s (cold)
Teacher-forced top-1 vs llama.cpp 189/192 short, 21/23 on 1425 tokens (misses are near-ties) -
Sampling, tool calls not supported supported

The llama.cpp numbers are from the first cold runs on this box and the RustingStrata numbers from the last runs before the model was deleted; they were not re-measured side by side, and both move with the page cache.

Greedy only. On an RTX 3060 12 GB with 30 GB RAM it decodes at about 2-3.6 tok/s (run to run it varies with the page cache, since the experts that miss VRAM are read from the GGUF), and prefill is one token at a time (about 0.5 s per token, 2 s per token at 1400 tokens). Context is capped at 4096 tokens. STRATA_DS_GPU_EXPERTS=<GB> changes the VRAM expert budget (10 worked on a 12 GB card). Output matches llama.cpp's greedy tokens up to near-ties: teacher-forced top-1 is 189/192 on three short prompts and 21/23 on a 1425-token prompt. Tool calls and sampling are not supported.

Tests

cargo test --release                      # GPU tests need the model: STRATA_MODEL=path/to/model.gguf
python -I tools/parity.py $M --extra "--gpu" --tag gpu   # agreement with llama.cpp (needs llama-server once)

Layout

src/gguf.rs (file reader), src/quant/ (dequantization), src/model.rs (CPU reference forward), src/cuda/ and src/vulkan/ (GPU kernels and shaders, behind src/backend.rs), src/gpu_model.rs (GPU forward, MTP, verify), src/gpu_model/prefill.rs (chunked prefill), src/expert_cache.rs (expert placement), src/cpu_moe.rs and src/cpu_worker.rs (CPU expert compute), src/serve.rs (HTTP), src/chat.rs (chat templates and tool calls), src/main.rs (CLI).

License

RustingStrata is released under the RustingStrata License 1.0. You may use, modify and distribute it, including for commercial services, as long as you keep the license and credit Vasyl Trefilov as the original author. You may not sell RustingStrata itself.