AEON-7/vllm-dflash

DFlash vLLM for DGX Spark
Plug & Play Block-Diffusion Speculative Decoding, optionally stacked with TurboQuant KV compression

docker pull ghcr.io/aeon-7/vllm-dflash:latest

Read First • Quick Start • Performance • TurboQuant • Which config? • Config • Troubleshooting

⚠ IMPORTANT — read first

Hardware requirements

Will NOT work on: H100/H200 (Hopper), A100 (Ampere), L40/L4, RTX 40-series, or any pre-Blackwell hardware. NVFP4 tensor cores are Blackwell-exclusive.

Critical caveats

1. First boot takes 5–10 minutes after the model is on disk (weight load + CUDA graph capture + FlashInfer NVFP4 autotune). Subsequent boots with cached JIT artifacts are ~2 min.
2. First inference request takes ~30 seconds even after the server reports healthy — vLLM does one-time CUDA graph specialization on the first real input. Always warm up 1–3 requests before measuring latency.
3. Model download is 20 GB; DFlash drafter auto-downloads another ~4 GB on first container start.
4. DFlash acceptance is content-dependent — expect 60 tok/s on code/reasoning, 30 tok/s on free prose. Random-token adversarial inputs will show no speedup.
5. Set VLLM_API_KEY as a persistent env var, not a one-shot $(openssl rand ...) expansion inside docker run — you need the same value to call the API later.
6. TurboQuant is optional. The default image does not include it. See TurboQuant (optional) for when to enable it.
7. Config sensitivity: the default tuned settings (MAX_NUM_SEQS=16, MAX_NUM_BATCHED_TOKENS=32768) are sized for DGX Spark's 128 GB. On smaller Blackwell cards (e.g. RTX 5090 32 GB) drop MAX_MODEL_LEN and GPU_MEMORY_UTILIZATION proportionally.

Before you file an issue

Check the Troubleshooting section — most problems are one of: queue saturation at high concurrency, CUDA OOM from oversized MAX_MODEL_LEN, or model-name mismatch between the mount and the API call.

Overview

A pre-built vLLM container tuned for NVIDIA DGX Spark (GB10 Blackwell), serving AEON-7/DFlash-Qwen3.5-27B-Uncensored-NVFP4 — a 27B hybrid linear-attention + full-attention model, NVFP4-quantized, vision-capable:

• DFlash block-diffusion speculative decoding (k=15) — 2–5× faster decode than vanilla vLLM depending on prompt class
• NVFP4 quantization with AWQ calibration — native Blackwell FP4 tensor cores
• OpenAI-compatible /v1/chat/completions, /v1/completions, /v1/models, /health
• Optional TurboQuant KV-cache compression for long-context / high-concurrency — see TurboQuant

All benchmarks below are on DGX Spark GB10 (128 GB unified memory, 273 GB/s LPDDR5X).

Quick Start

Three copy-paste steps. Plan for ~40 minutes end-to-end the first time — 20 min for model download (depends on your connection), 5–7 min for cold boot, a few seconds per warmup request.

1. Download the model (one-time, ~20 GB)

pip install "huggingface_hub[cli]"
huggingface-cli download AEON-7/DFlash-Qwen3.5-27B-Uncensored-NVFP4 \
    --local-dir /models/DFlash-Qwen3.5-27B-Uncensored-NVFP4

2. Launch the container

# Generate + remember an API key (save it — you'll need it for every request)
export VLLM_API_KEY=$(openssl rand -hex 32)
echo "VLLM_API_KEY=$VLLM_API_KEY" >> ~/.bashrc    # optional: persist across shells

docker run -d --name vllm-dflash \
    --gpus all --network host --ipc host --ulimit memlock=-1:-1 \
    -v /models/DFlash-Qwen3.5-27B-Uncensored-NVFP4:/models/target:ro \
    -e MODEL_PATH=/models/target \
    -e SERVED_MODEL_NAME=qwen35-dflash \
    -e DFLASH_DRAFTER=z-lab/Qwen3.5-27B-DFlash \
    -e DFLASH_NUM_SPEC_TOKENS=15 \
    -e MAX_MODEL_LEN=65536 \
    -e MAX_NUM_SEQS=16 \
    -e MAX_NUM_BATCHED_TOKENS=32768 \
    -e GPU_MEMORY_UTILIZATION=0.85 \
    -e ATTENTION_BACKEND=flash_attn \
    -e VLLM_API_KEY=$VLLM_API_KEY \
    ghcr.io/aeon-7/vllm-dflash:latest

The drafter (z-lab/Qwen3.5-27B-DFlash) auto-downloads on first run (~4 GB). Output tokens are addressed by the SERVED_MODEL_NAME you set above (qwen35-dflash here).

3. Wait for readiness, warm up, test

# Wait for healthy (cold start ~5–7 min — watch docker logs -f vllm-dflash if curious)
until curl -sf http://localhost:8000/health; do sleep 10; done
echo "server up"

# Warm up (first request does CUDA graph specialization — expect ~30 s)
for i in 1 2 3; do
  curl -sf -X POST http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $VLLM_API_KEY" \
    -d '{"model":"qwen35-dflash","messages":[{"role":"user","content":"hi"}],"max_tokens":16,"temperature":0}' \
    > /dev/null && echo "warmup $i ok"
done

# Real test
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $VLLM_API_KEY" \
    -d '{
        "model": "qwen35-dflash",
        "messages": [{"role": "user", "content": "Write a binary search in Python with type hints."}],
        "max_tokens": 256,
        "temperature": 0
    }'

You're running a 27B multimodal model with 2–5× speculative-decoding speedup on a 128 GB Spark.

Performance (DGX Spark GB10)

All numbers below are on unmodified ghcr.io/aeon-7/vllm-dflash:latest running AEON-7/DFlash-Qwen3.5-27B-Uncensored-NVFP4, DFlash k=15, 65K context, with the recommended configuration above. Measurements use natural-language prompts with temperature=0 for full determinism. See BENCHMARKS.md for the reproducible script.

Single-stream throughput by prompt style

Post-warmup, 3 runs, variance <0.3%.

DFlash acceptance length (tokens accepted per 15-token draft) ranges from ~2.0 on prose to ~5.5 on code. Per-position acceptance decays from ~78% at position 0 to <3% by position 8.

Concurrency scaling (natural prompts)

Code (best case for DFlash):

Prose (worst case):

At c=16 the container serves 328 tok/s on coding / 152 tok/s on prose, with TTFT below 900 ms.

Summary metrics

Which config should I use?

Short version: TurboQuant is a capacity unlock, not a throughput unlock. Enable it when you're memory-bound (many concurrent sessions or long contexts); skip it when you're compute-bound (few sessions, short prompts).

TurboQuant (optional)

For long-context or high-concurrency workloads, the container can be extended with 0xSero/turboquant KV-cache compression (4-bit keys, 3-bit values, Hadamard-rotation + Lloyd-Max codebooks — paper: arXiv:2504.19874).

TurboQuant is not enabled in the default image. To use it, build the extension Dockerfile in turboquant/ which pip-installs the plugin and wires it in via a Python .pth bootstrap.

Overhead vs baseline

Measured on the same model + tuned config. TurboQuant overhead is ~3% across all modes, concurrencies, and prompt styles — essentially free on short-to-medium outputs.

Code prompts

Prose prompts

Long-context behaviour

TurboQuant's hybrid-mode decode cost is flat until the 128-token ring buffer overflows, then grows with context length because each decode step has to dequantize more compressed history. Short-to-medium contexts see no penalty; decode slows measurably at 32K+.

When to enable TurboQuant

• Multi-session long-context serving — the real win is KV capacity, letting you hold more simultaneous sessions at full context (not visible in c=1 microbenchmarks)
• Agentic workloads with long rolling context where freeing compressed history recovers VRAM for the next request
• Any use case hitting OOM on long contexts under default KV

When NOT to enable TurboQuant

• Short-context single-user chat (<16K) — the decode overhead isn't worth the complexity when there's no capacity pressure
• Pure latency-critical 32K+ single-request paths — you'll eat the ~11% decode cost without the capacity payoff

Modes

Enabling

Build and run the TurboQuant variant:

cd turboquant
docker build -t vllm-dflash-tq:latest .

docker run -d --name vllm-dflash-tq \
    --gpus all --network host --ipc host --ulimit memlock=-1:-1 \
    -v /models/DFlash-Qwen3.5-27B-Uncensored-NVFP4:/models/target:ro \
    -e MODEL_PATH=/models/target \
    -e DFLASH_DRAFTER=z-lab/Qwen3.5-27B-DFlash \
    -e DFLASH_NUM_SPEC_TOKENS=15 \
    -e MAX_MODEL_LEN=65536 \
    -e MAX_NUM_SEQS=16 \
    -e MAX_NUM_BATCHED_TOKENS=32768 \
    -e GPU_MEMORY_UTILIZATION=0.85 \
    -e ATTENTION_BACKEND=flash_attn \
    -e ENABLE_TURBOQUANT=1 \
    -e TQ_MODE=hybrid \
    -e TQ_KEY_BITS=4 \
    -e TQ_VALUE_BITS=3 \
    vllm-dflash-tq:latest

Compatibility note

0xSero/turboquant currently requires a small patch to be CUDA-graph-safe (PR #12). The Dockerfile in turboquant/ applies that patch automatically. Once the PR is merged upstream, the patch step will be removed.

Configuration

Environment variables

TurboQuant-specific (when ENABLE_TURBOQUANT=1)

DGX Spark tuning recap

The three env vars that matter most on GB10:

MAX_NUM_SEQS=16                  # was 4 — unlocks c=8+ without queue saturation
MAX_NUM_BATCHED_TOKENS=32768     # was 8192 — matches scheduler's spec-decode headroom
GPU_MEMORY_UTILIZATION=0.85      # safe headroom; don't push higher on 128 GB unified

At defaults, c=8 hit TTFT p50 of 14.7 seconds due to queue saturation. With the tuned config, c=8 drops to 817 ms and c=16 becomes usable.

Troubleshooting

docker logs -f vllm-dflash

curl -sf http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY" | jq

How it works

DFlash — block-diffusion speculative decoding

DFlash speeds up generation by speculating multiple tokens per step using a small draft model, then verifying them against the target model in a single forward pass. The drafter here is a 5-layer Qwen3 variant fine-tuned to predict the next 15 tokens from the target's intermediate hidden states at layers (1, 16, 31, 46, 61).

Key properties:

• Lossless — every accepted token matches what greedy decoding would produce
• Memory-bandwidth-bound-friendly — a single target-model pass verifies many candidate tokens
• Content-adaptive — structured text (code, math) wins more than free prose

See paper: arXiv:2602.06036.

NVFP4 on Blackwell

NVIDIA's FP4 format (E2M1) is a native tensor-core datatype on Blackwell (B200, GB10, RTX 50×0). Unlike older INT4/GPTQ which introduce visible degradation, NVFP4 with AWQ_FULL calibration is effectively lossless. Our image autodetects NVFP4 checkpoints and routes through FlashInfer CUTLASS kernels.

Weights + activations are quantized; KV cache stays in BF16 by default (use TurboQuant to compress KV as well).

Hybrid architecture of Qwen3.5-27B

The model has 64 transformer layers arranged in a hybrid pattern:

• 48 linear-attention layers (Gated DeltaNet / Mamba-style recurrent state)
• 16 full-attention layers (classical attention with KV cache)
• 1 MTP head (used as the DFlash drafter anchor)

DFlash's target_layer_ids=[1,16,31,46,61] are the hidden-state checkpoints the drafter consumes. TurboQuant compresses only the 16 full-attention layers' KV cache; linear-attention layers have no K/V to compress (their recurrent state is already compact).

Why dense 27B beats 122B MoE on DGX Spark

DGX Spark is memory-bandwidth-bound (273 GB/s LPDDR5X unified). MoE experts require scatter/gather across the unified memory, which defeats the bandwidth budget. A dense 27B moves a predictable 20 GB of weights per token — ideal for the Spark's memory architecture. On coding/reasoning benchmarks it rivals or beats larger MoE variants that would OOM or thrash on this hardware.

Credits

• DFlash: Zheng et al., ICLR 2026 (arXiv:2602.06036)
• TurboQuant: Zandieh et al., ICLR 2026 (arXiv:2504.19874); this container uses 0xSero/turboquant as the plugin
• Model: AEON-7/DFlash-Qwen3.5-27B-Uncensored-NVFP4
• Drafter: z-lab/Qwen3.5-27B-DFlash
• vLLM: vllm-project/vllm 0.19.1

License

GPL-3.0 (inherited from 0xSero/turboquant when the TurboQuant extension is enabled; the base DFlash container is MIT). See LICENSE.

☕ Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development.

Patreon wrongfully shut down our account without notice and never responded to our appeals, so we built something even better. You can now support the work on LotusMake: buy us a Lotus Tea, fund GPU hours, back a project you want to see done, or subscribe for early and full access to publications while supporting what we build. You can become a creator there too: join, apply to become a creator, and get paid for the work you do in public.

AEON FORGE on LotusMake → lotusmake.com/c/aeon-forge · Subscribe for early and full access

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Subscribers also get early access to new releases, such as the AEON DFlash2 drafter.

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

₿ Bitcoin (BTC) bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4	Ξ Ethereum (ETH) 0x1512667F6D61454ad531d2E45C0a5d1fd82D0500
◎ Solana (SOL) DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t	ⓜ Monero (XMR) 836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.