Overview

Mixture-of-Experts (MoE) inference on CPU is bounded primarily by DRAM bandwidth per generated token. Standard 4.50-bpw scalar quantization (Q4_0) leaves significant memory-bandwidth headroom unused, while conventional 2–3 bit scalar K-quants (Q2_K, Q3_K) require multi-shift bit unpacking and fall back to Q4_0 whenever expert matrix dimensions are not multiples of 256 (such as 1856 × 2688 in Nemotron-3.5-30B-A3B).

llama-modchip v2.5.4 implements two block-64 quantization formats engineered for SIMD register lookup—Fractal-64 (GGML_TYPE_Q2_0 = 42, 30 bytes / 64 weights = 3.75 bpw) and LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43, 24 bytes / 64 weights = 3.00 bpw)—paired with cache-aware KV management (HARP-Cache), /dev/shm Slot-0 prefix snapshots, and RouteCrack + Stockfish-PVS v2 speculative decoding:

1. Domain-Aware LUT-VQ3-64 + Fractal-64 Hybrid Quantization: On uncompressed BF16 MoE weights (blk.1.ffn_up_exps.weight), Fractal-64 (3.75 bpw) reaches 17.75 dB SQNR (99.15% Cosine Similarity, +1.45 dB over Q3_K), while LUT-VQ3-64 (3.00 bpw) reaches 13.57 dB SQNR (97.78% Cosine Similarity, +3.42 dB over Q2_K) via 128 3D Lloyd-Max centroids decoded in registers with AVX-512 VBMI VPERMI2B. Keeping boundary layers and residual output projections (ffn_down_exps, MLA attention) in Fractal-64 while encoding interior expansion projections in LUT-VQ3-64 cuts up_exps footprint by 33.3% vs. Q4_0 with a +0.4818 (+6.92%) WikiText-2 Perplexity delta vs. True BF16 (7.4400 vs. 6.9582 on 12,288 wiki.test.raw tokens; +0.0790 / +1.07% vs. Fractal-64 7.3610), 0.1495 / 0.0944 nats/tok Logit KLD vs. True BF16 (78.1% Step-0 / 85.2% Matched Top-1, 98.0% Top-5), and 100.0 / 100.0 on benchmarks/eval_quality_and_tools.py.
2. DeepSeek-V4-Flash-0731 284B (13.17B Active) Inside 64 GB RAM: Four 3-shard 0731 checkpoints execute resident inside 64 GB DDR5 RAM: the donor-free safety-aligned default DeepSeek-V4-Flash-0731-MaxQ-Fractal (56.93 GiB disk / 57.54–58.13 GiB anonymous-heap VmRSS at -c 32768 -ub 512, single-pass 0731 @ 7872f01b BF16/FP8/MXFP4 weights across all 1,328 tensors: 174 dense bottlenecks [173 × Q6_K + 1 × Q8_0] + 317 Fractal-64 MLA/Indexer tensors + 129 routed-MoE tensors weighted by @antirez's 1.5M-token routed-moe-ds4-1p5m.dat [@ f71f23d552d6, calibrated 2026-05-12 on April V4-Flash chat-v2 activations], AtomicChat wiki-alt.txt ctx=5632 51-chunk Final estimate: PPL = 7.2977 ± 0.0504 [+60.8% vs 150.75 GiB MXFP4_MOE-BF16 4.5381, +22.2% vs 76.87 GiB UD-IQ1_S 5.9734, +14.4% vs 65.42 GiB AD-IQ1_M 6.3813; 12-chunk 7.3180 ± 0.1039], WikiText-2 PPL = 11.6139 ± 0.5845 [within < 0.15σ of Fractal-64 11.5288 ± 0.5563], 70.0 / 80 Functional Gate [30/30 Tools, 20/30 Code, 20/20 Needle; 8.66 tok/s pure N=1 / 8.80 tok/s Served Gate]), its opt-in uncensored twin DeepSeek-V4-Flash-0731-Abliterated-Fractal (56.93 GiB, --preset dsv4-ablit, identical single-pass recipe, WikiText-2 PPL = 11.3944 ± 0.5650, 70.0 / 80 Functional Gate [8.75 tok/s Served Gate], 0.0959 step-0 / 0.0512 nats/tok prefix-matched KLD vs MaxQ), compact safety-aligned DeepSeek-V4-Flash-0731-VQ3 (56.40 GiB, Unsloth UD-IQ1_S @ fbbb5b93 gate/up donor, wiki-alt.txt 12c PPL = 7.6579 ± 0.1077, WikiText-2 PPL = 12.2345 ± 0.5976, 64.0 / 80 Functional Gate, 9.05 tok/s N=1 / 9.15 tok/s Served Gate, 41.11 tok/s cold prefill, 19.4–25.3 tok/s agentic bursts), and cross-platform DeepSeek-V4-Flash-0731-Fractal (56.57 GiB, WikiText-2 PPL = 11.5288 ± 0.5563, 64.0 / 80 Functional Gate, 9.06 tok/s N=1 / 9.55 tok/s Served Gate).
3. Nemotron-3.5-30B-A3B (3.0B Active) in 16–24 GiB RAM: Safety-aligned NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64 (13.30 GiB, WikiText-2 PPL = 7.4400 ± 0.2626 vs. True BF16 6.9582 ± 0.2415) and Fractal-64 (14.36 GiB, PPL = 7.3610 ± 0.2602, 0.1254 / 0.0929 nats KLD vs. BF16) save -4.44 to -5.67 GiB RAM vs. Q4_0 at 24.18–25.02 tok/s Served Gate decode (+23.7% to +28.0% vs. Q4_0 19.54 tok/s, with up to 48.39 avg / 60.40 peak tok/s on Q4_0 in-context M=7 refactor bursts), alongside an opt-in uncensored Abliterated-VQ3-64 (13.30 GiB, PPL = 7.7721 ± 0.2835, 0.0308 / 0.0312 nats KLD & 87.5% Step-0 / 92.4% Matched Top-1 vs. VQ3-64, --preset nemotron-ablit).
4. Sub-20ms /dev/shm Slot-0 Restore + Cross-Platform Native Support: Restores the 40-tool Claude Code prefix (2,776 tokens) from /dev/shm in 10.4 ms (0.16s TTFT) on Linux, runs on Apple Silicon Metal (57.65 tok/s M1 Max, 78.60 tok/s M3 Max, 27.0 tok/s M5 Max 284B), and compiles natively on Windows 11 (MinGW-w64 UCRT64, verified up to 1M context).

[!NOTE] Operating Trade-Offs & Engine Constraints:

• Custom Tensor Types: Fractal-64 (GGML_TYPE_Q2_0 = 42, 30 B / 64 W) repurposes upstream GGML_TYPE_Q2_0 = 42 (18 B / 32 W) in-place and LUT-VQ3-64 adds GGML_TYPE_VQ3_0 = 43; vanilla llama.cpp, Ollama, vLLM, and LM Studio cannot load these GGUFs without the llama-modchip runtime.
• 64 GB Anonymous-Heap Margin (284B): --load-mode none --repack allocates 57.54–59.13 GiB in non-evictable anonymous heap (VmRSS). On 64 GB Linux hosts (~62.5 GiB usable RAM), use -c 32768 -ub 512 and enable zram or close heavy desktop browsers; reserve -c 262144 (256K context) for headless 64 GB nodes with swap/zram or 96 GB+ systems.
• Dual-Channel DDR5 Throughput Envelope (284B): On 2-channel DDR5-5600 (~75 GB/s sustained), raw N=1 unpruned Top-6 decode is 8.66–9.06 tok/s (8.80–9.55 tok/s in Served Gate decode) and cold prefill is 36.1–44.0 tok/s (30.7 tok/s at 3,038 tokens); higher sustained rates require multi-channel servers (EPYC / Threadripper) or Apple M5 Max (286.5 GB/s unified memory).

Signal Fidelity and End-to-End Benchmarks

1. Direct Signal Fidelity on Uncompressed BF16 MoE Weights

Measured via quantize_row_*_ref $\to$ dequantize_row_* on the uncompressed 59.0 GiB BF16 master checkpoint (blk.1.ffn_up_exps.weight, $N = 4,194,304$ weights, kurtosis $\kappa = 3.2–4.8$):

2. End-to-End Quality & Speed Ablations (benchmarks/eval_quality_and_tools.py + llama-bench)

Measured on 16-Core AMD Zen 5 (--cpu-range 0-15 --cpu-strict 1, Transparent HugePages enabled; full methodology and shipped-hybrid vs. full-VQ3 ablation breakdown in benchmarks/QUALITY_GATE_VQ3_64.md).

A. NVIDIA-Nemotron-3.5-30B-A3B (WikiText-2 PPL, 32-Probe / Multi-Step Logit KLD vs. True BF16, & 100-Point Gate)

(¹ WikiText-2 PPL [7.4400] and BF16/same-family Logit KLD [0.1495 / 0.0944, 0.0522] on LUT-VQ3-64 and Abliterated-VQ3-64 [7.7721, 0.0308] are measured directly on the shipped 13.30 GiB GGUFs; eval_nemotron_vq3_64.json [24.18 tok/s, 16.04 GiB VmRSS, 100/100 Gate] was recorded on the 13.08 GiB 23-layer --all-up-exps VQ3 ablation and serves as the architecture speed/VmRSS proxy for Abliterated-VQ3-64.)

B. DeepSeek-V4-Flash-0731 284B (284.33B Total / 13.17B Active, Unpruned Top-6 Routing)

(Methodology notes: [1] 284B Gate scores above use the uniform 80-point Functional Quality rubric across all variants; adding the 20-point speed/RAM rubric under a common --target-tps 9.0 target yields 89.56/100 for MaxQ-Fractal, 89.44/100 for Abliterated-Fractal, and 84.0/100 for Fractal-64 & VQ3. [2] In AtomicChat/dsv4-eval-artifacts, vendor numbers are Mean PPL(Q) from --kl-divergence on 8× RTX 5090 CUDA GPUs—where AtomicChat documents a 4–5% hardware gap [4.5381 RTX 5090 vs 4.3406 H100] on the same GGUF—whereas Synchopate 7.2977 ± 0.0504 is standalone llama-perplexity Final estimate: PPL on a 16C Zen 5 CPU; UD-IQ1_S [76.87 GiB] and AD-IQ1_M [65.42 GiB] exceed 64 GB host RAM. [3] On 284B, KLD against quantized MaxQ measures quantization-lineage divergence rather than ground-truth error, and prefix-matched rollout KLD stops at the first greedy token disagreement and therefore biases downward relative to unconditional 32-probe step-0 KLD.)

How LUT-VQ3-64 and Fractal-64 Work

1. 3D Sphere Packing (LUT-VQ3-64, GGML_TYPE_VQ3_0 = 43, 24 B / 64 W)

Scalar uniform quantizers quantize each coordinate $w_i \in [-M, +M]$ independently across the 3D Cartesian bounding cube $[-M, +M]^3$ of volume $8M^3$. Heavy-tailed MoE weights ($\kappa = 3.2–4.8$) have a radial joint density $p(\mathbf{w}) \propto \exp(-|\mathbf{w}|_p^\alpha / \sigma^\alpha)$ concentrated inside the inscribed $L_2$ sphere ($\text{Vol} \approx 4.189 M^3$, leaving $47.64%$ of the cube empty) or $L_1$ cross-polytope ($\text{Vol} \approx 1.333 M^3$, leaving $83.33%$ empty).

LUT-VQ3-64 places 128 3D Lloyd-Max centroids ($7\text{ bits} / 3\text{ weights} = 2.333\text{ bpw}$, antipodally symmetric $\mathbf{c}_{127-k} = -\mathbf{c}_k$) directly on the populated spherical/cross-polytope shell inside a 24-byte block_vq3_0 struct (patches/vq3_codebook.h):

• 2 bytes (d): FP16 super-block scale (scaled by GGML_VQ3_G = 8.6373016238e-3f).
• 22 bytes (qs[22], 176 bits):
  • qs[0..19] bits 0..6 (140 bits): 20 7-bit 3D indices covering 60 weights at stride 22 (j, j+22, j+44).
  • qs[20..21] bits 0..5 (12 bits): 2 6-bit 2D tail indices covering 4 weights (20, 42 and 21, 43).
  • s24 (24 bits packed in bit 7 of qs[0..21] and bit 6 of qs[20..21]): Eight 3-bit unsigned sub-scales ($m_b = 2 + u_b \in {2..9}$) across the eight 8-weight sub-blocks (24 bytes / 64 weights = 3.00 bpw).

2. Register-Resident AVX-512 VBMI VPERMI2B Lookup + Domain-Aware Layer Split

The 128-entry 3D codebook occupies $128 \times 3 = 384\text{ bytes}$, fitting inside 6 AVX-512 ZMM registers (cb0_lo/hi, cb1_lo/hi, cb2_lo/hi). AVX-512 VBMI VPERMI2B (_mm512_permutex2var_epi8) resolves 64 byte lookups per instruction directly in registers with zero L1/L2 cache reads. In the shipped Hugging Face checkpoints, interior pre-activation expansion tensors (ffn_up_exps in Nemotron, interior ffn_gate_shexp & ffn_up_shexp in DeepSeek-V4) use LUT-VQ3-64 (3.00 bpw), while boundary layers, ffn_down_* residual projections, and MLA attention tensors remain in Fractal-64 (GGML_TYPE_Q2_0 = 42, 3.75 bpw, 30 B / 64 W: 2B FP16 d + 4B 8×4-bit sub-scales {4..19} + 24B 3-bit indices into {-14,-8,-4,-1,+1,+4,+8,+14}).

Quantization and Memory Breakdown

Exact bit-widths (bpw), base checkpoints, and tensor allocations for each published GGUF:

Note on MoE Expert Routing (Unpruned Top-6 vs. Runtime Tail Pruning): By default (Unpruned Top-6), DeepSeek-V4-Flash evaluates all 6 routed experts + 1 shared expert (Top-6) per layer with zero runtime expert dropping relative to the loaded GGUF (1.00000 logit cosine similarity vs. unpruned Top-6 execution on the same quantized checkpoint; weight quantization fidelity itself is measured separately via BF16 SQNR and continuous WikiText-2 PPL / Logit KLD in benchmarks/QUALITY_GATE_VQ3_64.md). Runtime $L_2$-quadrature Top-5 / Top-4 pruning is isolated per user via 0700 DSV4_MOE_CFG_PATH ($XDG_RUNTIME_DIR/modchip-<uid>/dsv4_moe_cfg.bin) and resets to Top-6 on server startup.

Why DeepSeek-V4-Flash-0731 (and not V4.1-Flash-1017) for 64 GB Resident CPU Inference: DeepSeek-V4.1-Flash-1017 (DeepseekV41ForCausalLM, 510.3 GB source across 384 routed experts/layer) adds 188.83 GiB of native FP8 Engram lookup tables on top of 151.77 GiB of Q2 model weights (340.60 GiB total even at 2-bit quantization), requiring 128 GB+ RAM with NVMe SSD table streaming or 512 GB RAM for resident execution. DeepSeek-V4-Flash-0731 (284.33B total / 13.17B active, zero Engram tables) is the largest DeepSeek-V4 Flash checkpoint that runs 100% resident inside 64 GB RAM with zero disk streaming.

Quickstart

Linux & macOS (Automated Setup & Chat)

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

Direct Run (Portable GLIBC_2.30 Linux x86_64 -avx512 / -avx2 Binaries + Hugging Face .gguf)

Compatibility Note (GGML_TYPE_Q2_0 = 42 & GGML_TYPE_VQ3_0 = 43): Fractal-64 (30 B / 64 W) repurposes upstream GGML_TYPE_Q2_0 = 42 (18 B / 32 W) in-place and LUT-VQ3-64 (24 B / 64 W) appends custom type 43. Unpatched llama.cpp, Ollama, vLLM, and LM Studio will fail with unknown tensor type 43, a tensor bounds mismatch, or unknown model architecture: 'deepseek4'.

• Portable GLIBC_2.30 Linux x86_64 Binaries (GitHub Release v2.5.4): Built inside Ubuntu 20.04 (GLIBC_2.30 max symbol, -static-libstdc++, zero libssl/libcurl dependency; requires OpenMP libgomp1 / libgomp — Ubuntu 20.04+, Debian 11+, Fedora, RHEL/Rocky/Alma 9+) with SHA-256 verification in ./modchip download-bin and ./install.sh --prebuilt:
  • llama-server-modchip-linux-x86_64-avx512: AMD Zen 4/5 (Ryzen 7000/8000/9000, EPYC Genoa/Turin, Threadripper 7000) & Intel Ice Lake / Sapphire Rapids (avx512f + avx512vbmi; -DGGML_AVX512_BF16=OFF so Ice Lake / Tiger Lake never hit vdpbf16ps SIGILL).
  • llama-server-modchip-linux-x86_64-avx2: Pre-AVX512 x86_64 CPUs (x86-64-v3: Intel Haswell/Broadwell/Skylake/Alder/Raptor Lake, AMD Zen 1/2/3 — compiled with -mno-avx512f -DGGML_AVX_VNNI=OFF, 0 %zmm instructions). The unsuffixed llama-server-modchip-linux-x86_64 asset is an identical copy of this x86-64-v3 binary.

# Auto-select -avx512 or -avx2 binary for your Linux x86_64 CPU and verify SHA256:
BIN_ISA=$(grep -Eq "avx512_?vbmi" /proc/cpuinfo && echo "avx512" || echo "avx2")
curl -fLO "https://github.com/synchopate/llama-modchip/releases/download/v2.5.4/llama-server-modchip-linux-x86_64-${BIN_ISA}"
curl -fLO "https://github.com/synchopate/llama-modchip/releases/download/v2.5.4/SHA256SUMS"
sha256sum -c SHA256SUMS --ignore-missing
chmod +x "llama-server-modchip-linux-x86_64-${BIN_ISA}"

# 1A. Default 30B Nemotron-3.5-A3B VQ3-64 + Fractal-64 (13.30 GiB, Aligned — pass -c 32768 -ub 512 on 16 GB hosts):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64-CPU-GGUF \
  NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64.gguf --local-dir .
./llama-server-modchip-linux-x86_64-${BIN_ISA} -m NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64.gguf -c 262144 -t 16 -b 2048 -ub 512 --port 8080

# 1B. 30B Nemotron-3.5-A3B Fractal-64 (14.36 GiB, Aligned — recommended on AVX2-only CPUs & Apple Silicon):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF \
  NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .

# 2A. Default 284B DeepSeek-V4-Flash-0731-MaxQ-Fractal (56.93 GiB, Safety-Aligned Single-Pass Q6_K + Fractal-64):
hf download Synchopate/DeepSeek-V4-Flash-0731-MaxQ-Fractal-64GB-CPU-GGUF \
  --include "DeepSeek-V4-Flash-0731-MaxQ-Fractal-*.gguf" --local-dir ./dsv4-maxq
# Safe 64 GB Desktop default (-c 32768 -ub 512 -> ~57.5–58.1 GiB anonymous-heap VmRSS):
./llama-server-modchip-linux-x86_64-${BIN_ISA} -m ./dsv4-maxq/DeepSeek-V4-Flash-0731-MaxQ-Fractal-00001-of-00003.gguf \
  -c 32768 -t 16 -b 2048 -ub 512 -ctk q8_0 -ctv q8_0 -fa on --port 8080
# Dedicated Headless 64 GB Server (with zram/swap) or 96 GB+ Workstation (full 256K MLA context):
# ./llama-server-modchip-linux-x86_64-${BIN_ISA} -m ./dsv4-maxq/DeepSeek-V4-Flash-0731-MaxQ-Fractal-00001-of-00003.gguf \
#   -c 262144 -t 16 -b 2048 -ub 512 -ctk q8_0 -ctv q8_0 -fa on --port 8080

# 2B. Compact Safety-Aligned 284B DeepSeek-V4-Flash VQ3 (56.40 GiB) or Cross-Platform Fractal-64 (56.57 GiB):
hf download Synchopate/DeepSeek-V4-Flash-0731-VQ3-64GB-CPU-GGUF \
  --include "DeepSeek-V4-Flash-0731-VQ3-*.gguf" --local-dir ./dsv4-vq3
hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF \
  --include "DeepSeek-V4-Flash-0731-Fractal-*.gguf" --local-dir ./dsv4

# 2C. Opt-In Abliterated / Uncensored Variants (284B dsv4-ablit & 30B nemotron-ablit — requires confirmation):
# ./install.sh --preset dsv4-ablit        # Synchopate/DeepSeek-V4-Flash-0731-Abliterated-Fractal-64GB-CPU-GGUF (56.93 GiB)
# ./install.sh --preset nemotron-ablit    # Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Abliterated-VQ3-64-CPU-GGUF (13.30 GiB)

[!CAUTION] Responsible-Use Disclaimer (Opt-In dsv4-ablit & nemotron-ablit Checkpoints): Both optional abliterated checkpoints (DeepSeek-V4-Flash-0731-Abliterated-Fractal and NVIDIA-Nemotron-3.5-30B-A3B-Abliterated-VQ3-64) derive from base models in which refusal-mediating activation directions were removed via orthogonal projection. Standard upstream refusal filters are disabled, and ./install.sh never selects them without explicit interactive [y/N] confirmation (or --accept-abliterated in non-interactive mode).

• Intended Scope: Authorized security research, red-teaming, CTF competitions, vulnerability auditing, and local offline software engineering only.
• Deployment: Do not expose an unmoderated endpoint of an abliterated checkpoint to public or multi-user networks without external access control and safety filtering.
• Safety-Aligned Defaults: Default ./modchip setup installs safety-aligned dsv4-maxq (60 GB+ / 64 GB class) or nemotron-vq3 / nemotron-fractal (16–59 GB).

64 GB Host RAM Headroom (-c 32768 -ub 512 Desktop vs. -c 262144 Headless): Because --load-mode none --repack stores repacked CPU weights in anonymous heap (57.54–59.13 GiB VmRSS), the Linux kernel cannot demand-page those pages back from disk under desktop RAM pressure without swap/zram. On a 64 GB Linux box (~62.5 GiB usable RAM), keep -c 32768 -ub 512 (start_server.sh enforces -c 32768 -ub 512 and disk-backed slot snapshots automatically on <= 68 GB hosts unless MODCHIP_CTX_SIZE is set) and enable zram or close heavy GUI browsers. Use -c 262144 (256K tokens) on dedicated headless 64 GB nodes with zram/swap or 96 GB+ workstations. Measured context scaling up to 3,038 tokens (30.7–38.4 tok/s prefill, 9.16 tok/s decode) is reported in docs/DEEPSEEK_V4_OPTIMIZATIONS.md §6.

Windows 10/11 (Patched VQ3-64 / Fractal-64 Engine & Hybrid GPU+CPU Pack)

• Standard GGUFs (Q4_0, MXFP4, Q3_K_M, Q8_0) with Automatic GPU+CPU Offload: Run windows/AVVIA_MODCHIP.bat (windows/README.md). It detects physical CPU cores, free RAM, and GPU VRAM (nvidia-smi), downloads official llama.cpp CUDA 12.4 / Vulkan binaries, computes --n-cpu-moe, and starts http://127.0.0.1:8080.
• Patched DeepSeek-V4 284B & Nemotron-30B (Fractal-64 type 42 & LUT-VQ3-64 type 43): Run powershell -ExecutionPolicy Bypass -File windows\build_patched_engine.ps1 -Target dsv4 (or -Target nemotron) to compile the native MinGW-w64 UCRT64 AVX-512 engine, then launch via windows/AVVIA_DSV4.bat (details in docs/WINDOWS_PORT.md).

Hardware Readiness Check (Zero Model Download Required)

curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3

Submit your terminal scorecard via a GitHub Benchmark Issue or GitHub Discussions to add your CPU to the leaderboard below.

Supported Models and Model-Specific Optimizations

Each model family has a dedicated technical guide with mathematical derivations, operator-level latency profiles, and quality gate benchmarks:

Supported Chips and Processor-Specific Optimizations

Microarchitectural details and tuning flags for each processor and GPU hybrid are documented in docs/HARDWARE_OPTIMIZATIONS.md:

Headline Benchmarks: Pure CPU & Hybrid Summary

1. DeepSeek-V4-Flash 284B (13.17B Active) on a 64 GB Zen 5 CPU (AMD Ryzen 9 9955HX, 0 GPUs)

Operator-level profiling and mathematical derivations in docs/DEEPSEEK_V4_OPTIMIZATIONS.md (probabilities and KL-divergence measured relative to unpatched Top-6 execution on the 56.57 GiB Fractal checkpoint):

2. NVIDIA-Nemotron-3.5-30B-A3B (3.0B Active) at 256K Context (AMD Ryzen 9 9955HX, 0 GPUs)

Full ablation tables and 100/100 Quality Gate reports in docs/NEMOTRON_OPTIMIZATIONS.md and benchmarks/QUALITY_GATE_VQ3_64.md:

3. DeepSeek-V4-Flash 284B (13.17B Active) on Apple Silicon (Apple M5 Max, 64 GB Unified Memory)

• Prefill Throughput (pp512): 169.4 ± 29.6 tok/s (Apple Metal accelerated)
• Decode Throughput (tg128): 27.0 ± 0.4 tok/s in llama-bench
• Real Served Generation (llama-server): 20.7 – 23.6 tok/s across coding and prose (code_edit: 20.7 tok/s, code_new: 23.3 tok/s, free_text: 23.6 tok/s) with 0 swapouts (iogpu.wired_limit_mb=57344)
• Architecture & Placement: Apple M5 Max (18-Core CPU + 40-Core Metal GPU), 64 GB LPDDR5X. 5 pure-IQ1_S MoE layers offloaded to CPU (blk.(10|12|14|16|18).ffn_.*_exps.=CPU), remainder on Metal.
• Driver & Patches: patches/apple-silicon/0001 (Fractal-64 Q2_0 NEON/Metal), 0002 (Metal tensor run mapping), 0003 (Metal register unpack). Built via apple-silicon/build-dsv4-engine.sh and launched with apple-silicon/run-dsv4.sh. Contributed by Bobby Grayson (@notactuallytreyanastasio). Full report: apple-silicon/results/dsv4-m5max.md.

Community CPU and Hybrid Leaderboard

Run ./modchip bench (or windows\BENCHMARK_PC.bat on Windows) and Submit Your Scorecard (GitHub Issue) (pure-CPU entries sorted by Refactor decode):

llama-modchip/
├── modchip                                    # Unified CLI (setup | start | chat | bench | eval | quantize | web | claude | stop)
├── install.sh                                 # Automated Installer (Linux)
├── README.md / ARCHITECTURE.md / FAQ.md       # Overview, Mathematical Architecture & Engineering FAQ
├── RELEASE_NOTES.md                           # v2.5.4 Release Notes
├── docs/
│   ├── NEMOTRON_OPTIMIZATIONS.md              # Nemotron-3.5-30B-A3B & Nemotron-H-8B optimizations
│   ├── DEEPSEEK_V4_OPTIMIZATIONS.md           # DeepSeek-V4-Flash 284B profiling, SVD & L2-quadrature MoE
│   ├── HARDWARE_OPTIMIZATIONS.md              # Chip-by-chip optimizations (Zen 5, Threadripper, Vulkan, CUDA, Metal, ARM64)
│   └── WINDOWS_PORT.md                        # Native Windows 11 patched-engine build & Ryzen 7 9800X3D 1M-context report
├── vq3/                                       # LUT-VQ3-64 3D Lloyd-Max codebook, self-contained C encoder & kernel tests
├── patches/
│   ├── apple-silicon/                         # Apple Silicon engine patches (Fractal-64 Q2_0 NEON/Metal, register unpack)
│   ├── deepseek-v4-flash-fractal-256k.patch   # Base engine patch (DeepSeek-V4-Flash 284B + Fractal-64 Q2_0 = 42)
│   ├── dsv4-64gb-cpu.patch                    # Apple Silicon base engine patch (pinned to llama.cpp d2f83055)
│   ├── dsv4-ipc-hardening.patch               # Per-user 0700 IPC & /dev/shm_<uid> hardening patch (Linux/macOS/Windows)
│   ├── lut-vq3-64-avx512-vbmi.patch           # Incremental LUT-VQ3-64 (VQ3_0 = 43) AVX-512 VBMI + AVX2 + scalar patch
│   ├── vq3_codebook.h                         # 128-entry 3D + 64-entry 2D LUT-VQ3-64 codebook header
│   └── zen5-nemotron-256k-optimizations.patch # Standalone ggml/llama core patch for Nemotron-3.5
├── windows/                                   # Windows 10/11 Patched Engine Builder (build_patched_engine.ps1, AVVIA_DSV4.bat) & Hybrid Pack
├── apple-silicon/                             # Apple Silicon M1–M5 setup scripts (build-dsv4-engine.sh, run-dsv4.sh, M5 Max report)
├── fedora/                                    # Fedora 44 / AMD Ryzen 7 8745HS (GEEKOM A8) all-in-one script
├── speculative/                               # RouteCrack + Stockfish-PVS v2 (rc_stockfish.h, ngram-map.cpp)
├── benchmarks/                                # eval_quality_and_tools.py (100-pt Quality Gate), QUALITY_GATE_VQ3_64.md & results/
├── gateway/sse_gateway.py                     # Agentic SSE Proxy + Compact Tool Table + Dark-Mode Web Chat HUD (:8787)
└── scripts/                                   # build_zen5_llama.sh, build_{nemotron,dsv4}_vq3.py, build_fractal_moe_gguf.sh, chat.py

Acknowledgments

• Salvatore Sanfilippo (@antirez) for single-pass DeepSeek-V4 routed-MoE quantization research and the 1.5M-token DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.dat (antirez/deepseek-v4-gguf @ f71f23d552d6, committed 2026-05-12 from April DeepSeek-V4-Flash chat-v2 activations) importance matrix used in the donor-free single-pass MaxQ-Fractal (v2.5.3) and Abliterated-Fractal checkpoints.
• Unsloth (unsloth/DeepSeek-V4-Flash-0731-GGUF) for the UD-IQ1_S (@ fbbb5b93fb787c21338159b0af3318bb3f4d9768) checkpoint used as the upstream donor (86 routed-MoE gate_exps/up_exps tensors across 43 layers: 70 in blk.0..34 [60 × IQ1_S + 10 × IQ2_XXS in anchor layers {1,4,26,28}] copied byte-for-byte + 16 in blk.35..42 transcoded) when building the compact DeepSeek-V4-Flash-0731-Fractal (56.57 GiB) and DeepSeek-V4-Flash-0731-VQ3 (56.40 GiB) variants.
• AtomicChat (AtomicChat/dsv4-eval-artifacts) for publishing the open wiki-alt.txt (-c 5632 -b 5632) evaluation corpus and cross-vendor DeepSeek-V4-Flash-0731 perplexity reference logs.
• DeepSeek AI (deepseek-ai) and NVIDIA for the foundational DeepSeek-V4-Flash (MIT License) and Nemotron-3.5-30B-A3B (OpenMDW-1.1) architectures, and ggml-org/llama.cpp for the upstream C/C++ inference runtime.
• Prem Research (prem-research) and HangGlidersRule for the DeepSeek-V4-Flash-0731-abliterated and Darkstar-Nemotron-3.5-Lightning-30B-A3B-Abliterated-BF16 checkpoints used for the opt-in dsv4-ablit and nemotron-ablit presets.