defai-digital/ax-engine
defai-digital/ax-engine has 34 GitHub stars and 2 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 17:44 UTC. Last push 2026-11-13.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Real speed. Multi-model LLM serving with prefix reuse, MTP acceleration — built for Apple Silicon
AIREADME.md
OriginalAX Engine
AX Engine is a Mac-first Apple Silicon inference runtime optimized first for Qwen 3.8 27B AXQ. The Qwen 3.8 27B and Flash Next product scope is MLX + AXQ MXFP4 / MXFP8 with MTP sidecars. The available pinned product selectors load MXFP4 MTP packs; MXFP8 pack admission and qualification are pending. Standalone affine 4/6/8-bit and non-MTP variants for these two models are retired. MTP execution remains explicit opt-in until its gates pass. Install the macOS app with one curl command to download models, chat and serve OpenAI-compatible endpoints locally.
Primary optimization target. Checkpoint Tier 1. MTP Tier 2 pending. AX certification record: Candidate (gates open).
The default pack is qwen3.8-27b:axq
(AutomatosX/AX-Qwen3.8-27B-MLX-AXQ-MXFP4-MTP
@ b2c5354f779e430d0c1733143db848a72b71c16e). Additional Qwen, Gemma, GLM, and
other certified families stay supported; they are not the first-run or
qualification center. Super-class Qwen 3.8 (2.4T) is experimental only.
AX Code's managed local default is not this 27B Qwen alias. The coding harness selects Tiel Coder 35B A3B MXFP4 MTP, with Cyber-Tiel as the alternate pack. Do not mix the Qwen 27B qualification table with the Tiel campaign.
NVIDIA/CUDA fleet serving lives in AX Serving. AX Engine remains the local Apple Silicon runtime and no longer ships the former vLLM or TensorRT provider bridges, runtime package, container, or CUDA qualification scripts.
Browse AutomatosX serve-ready chat / coding / embedding snapshots in the AutomatosX model collection on Hugging Face. Additional native families (GLM 4.7 Flash, Nemotron Omni, Unlimited-OCR, Whisper, MiniCPM-V, and others) are documented under Supported Models.
Requires macOS 26 (Tahoe)+ on Apple Silicon. Hardware tiers:
| Tier | Chips | What to expect |
|---|---|---|
| Primary | M5 and later | Defaults, tuning and published performance are measured here. Best experience. |
| Supported | M3, M4 | Tested for correctness every release; may be slower than M5. |
| Best-effort | M2 | Runs, but is not tuned or qualified. |
| Not supported | M1 | Startup fails closed. |
Product SKUs:
- Qwen 3.8 27B AXQ (
qwen3.8-27b:axq) — optimized on M5-class Macs (32 GB+ recommended). The existing qualification record was measured on a Mac mini M4 Pro 64 GB, a supported M4-tier configuration; MXFP4 SKU qualification pending - MacBook Pro M5 Max 128 GB — qualification and certification standard for Qwen 3.8 Flash Next MXFP4 MTP
(125B-A6B). Second SKU. Qualification and certification standard: MacBook Pro M5 Max 128 GB, single-stream resident (one request at a time, MTP off, contexts up to 32k). MXFP4 MTP target; native functional controls verified; direct-default release gates passed on this standard host; model MTP is an explicit experimental opt-in. MTP Tier 2 pending. AX certification record: Candidate (direct-default release gates passed on MacBook Pro M5 Max 128 GB; MTP gates open).
qwen3.8-flash-next:axqandqwen3.8-flash-next:mxfp4select the pinned MXFP4 MTP pack. Start the server with--max-concurrent-requests 1. Measured decode on one machine: 32 / 32 / 28 tok/s at 2k / 8k / 32k context. Historical 6-bit numbers do not qualify MXFP4. This host is also the benchmark host for other packs. - Mac Studio, Ultra-class Apple Silicon with 192 GB+ RAM (M3 Ultra or newer; M2 Ultra is best-effort) — larger-memory host for the same Flash Next pack. The 2026-10-05 direct-default release gates were measured here and stay bound to this host.
Compact single models (Qwen 3.5 9B 4-bit preferred) still fit 16 GB. Prefer 4-bit for headroom on that class.
Wired-memory controls are available on every Mac AX Engine runs on (M2 and newer); they are not M5-only. The automatic no-wire optimization currently targets only the two audited Tiel MXFP4 MTP exports on M5 Max with at least 128 GiB. Other supported hosts retain existing wiring. Model fit and measured speed still depend on memory capacity and workload; see the Tiel residency policy.
Why AX Engine
- MLX-backed AXQ MX packs —
qwen3.8-27b:axqselects the pinned MXFP4 MTP pack. Flash Next has its ownqwen4_exproute and MXFP4 MTP selector. MXFP8 is in scope with pack admission pending. See the format and migration policy. - DFlash2 on eligible greedy Qwen text — Qwen 3.8 27B and Qwen 3.6
35B-A3B draft a block with DFlash2 when the matching Apache-2.0 checkpoint
is already on disk (
z-lab/Qwen3.8-27B-DFlash2,incoai/Qwen3.6-35B-A3B-DFlash2). The server does not download it, and it is not inside the AXQ pack. Sampled requests, grammars, images, and a missing checkpoint stay on the normal routes. MTP stays explicit opt-in. See DFlash2. - NAX attention on M5 — on M5 or later with macOS 26.2 or newer, MLX
selects Neural Accelerator GEMM and SDPA. AX keeps native causal attention
for allowlisted full-attention prefill of at least 1024 tokens, including
Qwen 3.8 27B. Gemma stays off that policy. M3 and M4 do not arm it.
AX_MLX_NAX_ATTENTION=0turns the attention policy off. See NAX attention. - Evidence tied to the pack — historical 6-bit benchmark results remain published below and do not describe the replacement default. MXFP4 and MXFP8 need their own runtime, quality, memory and MTP evidence.
- Multi-model on one process — keep a scoped set of Qwen 3.5/3.6,
Qwen3-Coder-Next, Gemma 4, and embedding models resident (
load_mode=add), route by requestmodel(chat + embeddings together), with fair Metal turn arbitration, memory preflight, and optional idle eviction. Exact-prompt prefix reuse is the S1 differentiator: official dual-model campaign (Qwen stream + Gemma 13.8k prefill) clears all locked gates at 5.03× median throughput vs a multi-process peer MLX server — see Performance and Server: Multi-model - You own the stack you serve — AX runs the MLX graph, KV/runtime, and
OpenAI-compatible server for supported Gemma / Qwen / GLM (and other direct
families);
mlx-lmandllama.cppstay optional compatibility adapters - Engine-owned scheduling and KV — continuous batched decode
(certification-gated), chunked prefill, preempt-and-recompute, and a paged
KV ledger with cross-request prefix sharing run inside the engine, with a
published
ax_runtime_*saturation contract for fleet routers — see Scheduling and KV runtime - Native media and speech — image/video chat, mixed image+audio reasoning, OCR, and Whisper transcription/translation run through repo-owned MLX graphs with capability-gated OpenAI endpoints (checkpoint-authoritative; see media table below)
- Clear fleet boundary — AX Engine owns local Apple Silicon inference; AX Serving owns fleet orchestration and NVIDIA/CUDA workers
- Claims you can audit — public rows ship with checked-in artifacts (route, model snapshot, sampler, accept rate, provenance)
Components and workflow
Clients enter through the HTTP server, the Python binding, or a language SDK.
ax-engine-sdk keeps one session per loaded model on that model's worker
thread. ax-engine-core admits the request, plans a decode-first batch, and
owns the paged KV ledger. ax-engine-mlx runs the graph, including DFlash2
block drafting for eligible greedy Qwen text, MTP, n-gram acceleration,
expert paging, and NAX attention on M5. Optional mlx_lm.server and llama.cpp
adapters stay on the compatibility route.
Normative detail is in Architecture.
Install AX Engine for macOS
Use the curl installer on Apple Silicon (M5 or later recommended; M3–M4 supported; M2 best-effort), macOS 26.2 or newer. It installs AX Engine.app with its native inference runtime bundled. You do not need Homebrew, Python, Xcode or a Hugging Face account for public models. Quit AX Engine before installing or upgrading, then run in Terminal:
curl -fsSL https://download.automatosx.com/ax-engine-desktop/install | bash
The installer verifies signed release metadata, the download and Apple's app
identity before installing /Applications/AX Engine.app. macOS may request your
password if that directory needs administrator access. It preserves models,
settings and conversations, and restores the previous app if replacement fails.
It does not install a global ax-engine command on your shell's PATH.
Open AX Engine from Applications. In Models → Catalog, choose a model that fits your Mac and download it; use the app's server controls and streaming chat after the model is ready. Model downloads are separate from installation. The default Qwen 3.8 27B pack is about 17.4 GB; allow about 20 GB of free disk space and prefer 32 GB or more of memory for it. Catalog inclusion does not establish model qualification or memory fit.
Update or verify the app
Quit AX Engine and run the same curl command again to update to the latest stable GUI release. Automatic app updates are not available yet.
To download and verify the release without installing it:
curl -fsSL https://download.automatosx.com/ax-engine-desktop/install | bash -s -- --verify-only
For an existing user-writable Applications directory, pass
--applications-dir "$HOME/Applications" after bash -s --. Run with --help
for installer options. If the download fails, stop and retry later; do not
remove quarantine or disable Gatekeeper. See the macOS app guide
for usage and distribution guide for release verification.
Advanced: CLI and Python installation
The curl command above is the end-user app installation path. The channels below are for shell automation, direct API serving or Python development; they do not install the GUI app. Native CLI packages require Apple Silicon (M5 or later recommended; M3–M4 supported; M2 best-effort) and macOS 26 or newer.
| Goal | Channel |
|---|---|
| Python SDK and CLI in one environment | pip |
| CLI/server without Homebrew or the Python SDK | Standalone archive |
| CLI/server managed by Homebrew | Homebrew |
Install with pip (no Homebrew needed)
Install Python from the official macOS installer: open python.org/downloads/macos, download the Python 3.13 installer (any 3.13.x), run it, then quit and reopen Terminal. Python 3.12 through 3.14 work; 3.15 does not yet work with the
downloadextra because MLX publishes no 3.15 wheels.Check the version. Use
python3.13explicitly: thepython3that ships with macOS is 3.9, cannot install AX Engine, and may open an Xcode Command Line Tools prompt.python3.13 --version # must print: Python 3.13.xCreate an environment and install:
python3.13 -m venv "$HOME/.venvs/ax-engine" source "$HOME/.venvs/ax-engine/bin/activate" python -m pip install --upgrade pip python -m pip install --upgrade "ax-engine[download]>=8.0.7,<9" ax-engine doctorMake
ax-engineavailable in every new Terminal (otherwise it is only found after running thesourceline above):echo 'export PATH="$HOME/.venvs/ax-engine/bin:$PATH"' >> ~/.zprofile
The wheel provides the Python package (import ax_engine), the ax-engine and
ax-engine-server commands, and the bench binary used by diagnostics. It
bundles the MLX runtime; Xcode and the Metal Toolchain are not needed.
pip install printing No matching distribution found almost always means an
old Python (3.9 is the usual culprit) or macOS older than 26 — see
Python installation troubleshooting.
Standalone download (no Homebrew, no Python SDK)
A signed and notarized arm64 tarball with the ax-engine, ax-engine-server
and ax-engine-bench binaries and the MLX runtime. Downloading with curl
avoids the browser quarantine prompt:
TAG=$(curl -fsSLI -o /dev/null -w '%{url_effective}' \
https://github.com/defai-digital/ax-engine/releases/latest | sed 's|.*/||')
ASSET="ax-engine-$TAG-macos-arm64.tar.gz"
BASE="https://github.com/defai-digital/ax-engine/releases/download/$TAG"
cd "$(mktemp -d)"
curl -fsSLO "$BASE/$ASSET" && curl -fsSLO "$BASE/$ASSET.sha256"
shasum -a 256 -c "$ASSET.sha256" # must print: OK
mkdir -p "$HOME/.local/ax-engine" && tar -xzf "$ASSET" -C "$HOME/.local/ax-engine"
echo 'export PATH="$HOME/.local/ax-engine:$PATH"' >> ~/.zprofile
export PATH="$HOME/.local/ax-engine:$PATH"
ax-engine doctor
Serving a model directory you already have needs nothing else. Downloading
models by alias (ax-engine serve qwen3.8-27b:axq) also needs Python 3.12+ and
huggingface-hub: follow
Model download helpers.
Homebrew
brew install defai-digital/tap/ax-engine
ax-engine doctor
The self-contained release formula installs the release's pinned
libmlx.dylib, libjaccl.dylib, and precompiled mlx.metallib; it does not
build MLX from source. Native inference from a ready local model directory
needs no Python, Xcode, or Metal Toolchain. Model aliases and preparation
helpers use Python 3.12+; online downloads also require huggingface-hub.
Follow the
model download helpers setup
before downloading or serving an alias.
If more than one install channel is present, which -a ax-engine lists every
copy; keep one channel per shell. See Getting Started
for the full channel comparison and troubleshooting.
Run the CLI server
Use AX Engine for macOS for graphical model downloads, resource charts, server controls and streaming chat. The terminal UI has been retired; the CLI remains available for automation and local model paths.
Serve Qwen 3.8 27B AXQ MXFP4 MTP, then request from another
terminal. qwen3.8-27b:axq is the pinned AutomatosX MXFP4 MTP pack (same
checkpoint as ax-qwen3.8-27b). The command reuses the cached snapshot when
present and downloads it otherwise (about 17.4 GB; the first start waits for
the download). Listen defaults to 127.0.0.1:31418:
ax-engine serve qwen3.8-27b:axq
curl http://127.0.0.1:31418/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":64}'
Option C — coding model (resolve + serve; Tiel Coder 35B-A3B is a development MXFP4 MTP pack, about 22 GB):
ax-engine serve tiel-coder-35b:axq --port 31418
If a download stops with Repository Not Found or 401, that snapshot is not
publicly downloadable. Pick one from the live
AutomatosX collection;
qwen3.8-27b:axq above is public.
Python wheel, source builds, and troubleshooting: Getting Started.
Models
Public AutomatosX model downloads
ax-engine download --list prints the curated public
AutomatosX model collection.
The GUI shows the runtime catalog in Models → Catalog, completed models in
Your Models, and current downloads in Downloads. Downloads are public,
unrestricted AutomatosX packs only; no Hugging Face account or token is needed.
Download other models independently and use the CLI with a local model path.
Local models remain accessible when the Hub is unavailable. Qwen 3.8 27B AXQ (MXFP4 MTP pack),
Qwen 3.6, Qwen 3.5, and Gemma 4 variants published there (plain 4-bit/6-bit,
QAT, OptiQ, AXQ where available) are first-class serve targets. Other native families (for example GLM 4.7 Flash, Nemotron
Omni, Unlimited-OCR, Whisper, MiniCPM-V) use the repo-owned runtime via serve
aliases, presets, or manual model directories; they are not all AutomatosX-managed
packages. Full matrix:
Supported Models.
Qwen 3.8 Super-class (2.4T) is experimental only, not a production target.
Those packs can technically load through the SSD expert-stream path
(--stream-experts, default auto), but local inference is too slow even at
2-bit to recommend or certify. Start local serving on Qwen 3.8 27B AXQ
MXFP4 MTP (qwen3.8-27b:axq).
Recommended starting packages (serve-ready, match published benches):
| Goal | Alias / family | Why |
|---|---|---|
| Default dense chat + MTP | qwen3.8-27b:axq (pinned AXQ MXFP4 MTP) |
Qwen 3.8 27B; MXFP4 with MTP sidecar, qualification pending |
| Fastest MoE chat + MTP | ax-qwen3.6-35b-a3b (4-bit or 6-bit MTP) |
Strongest serving and MTP peer decode rows |
| Dense chat + MTP (3.6) | ax-qwen3.6-27b (6-bit MTP preferred) |
High same-package MTP speedup; solid serving |
| AXQ evaluation candidate | qwen3.6-27b:axq (pinned 6-bit) |
Qwen 3.6 27B AXQ candidate; explicit until its checkpoint certification gates pass |
| Vision MoE Instruct AXQ | ax-qwen3-vl-30b / ax-qwen3-vl-30b-4bit |
Qwen3-VL 30B-A3B Instruct AXQ packs; candidate, no MTP |
| Holo3 GUI-agent AXQ | holo3-35b / holo3-35b:axq |
Qwen3.5-class 35B-A3B MoE; Tier 1 certified text path; no MTP |
| Ornith coding AXQ | ornith-35b / ornith-35b:axq |
Qwen3.5-class 35B-A3B MoE coding agent; Tier 1 AXQ; no MTP |
| Ornith 1.5 MTP | ornith-1.5-35b:axq |
Qwen3.5-class 35B-A3B MoE with HF per-expert MTP sidecar; development pack |
| Tiel coding MTP | tiel-coder-35b:axq / cyber-tiel-coder-35b:axq |
Qwen3.5-class 35B-A3B MoE MXFP4 packs; native HF per-expert MTP mode; development, unqualified |
| GPT-OSS AXQ | gpt-oss-20b:axq / gpt-oss-120b:axq |
AutomatosX AXQ; bare aliases stay mlx-community MXFP4-Q4 |
| Nemotron 3 Nano AXQ | nemotron-3-nano / nemotron-3-nano:axq |
nemotron_h 30B-A3B; development AXQ |
| Muse-Glimmer image-text AXQ | muse-glimmer-30b / muse-glimmer-30b:axq |
Meta dense 30B image-text agent; ATEM chat; development AXQ; no MTP |
| Multimodal chat + MTP | ax-gemma4-12b / 26B / 31B Assistant-MTP |
Image/audio/video + assistant draft package |
| Coding agent | ax-qwen3-coder-next |
Coding-focused MoE; multi-model friendly |
| Embeddings | ax-embeddinggemma-300m or Qwen3-Embedding aliases |
Batched ingest scale in full results |
Repositories ending in -MTP or -Assistant-MTP already contain the prepared
sidecar or assistant artifacts and model-manifest.json. Download them with
the standard flow; do not run download-mtp afterward.
| Family | Role | Supported AutomatosX snapshots |
|---|---|---|
| Qwen 3.5 9B | Chat / agent | AX-Qwen3.5-9B-MLX-4bit-MTPAX-Qwen3.5-9B-MLX-6bit-MTPAX-Qwen3.5-9B-MLX-OptiQ-4bit-MTP |
| Qwen 3.6 27B | Chat / agent / multimodal | AX-Qwen3.6-27B-MLX-4bit-MTPAX-Qwen3.6-27B-MLX-6bit-MTPAX-Qwen3.6-27B-MLX-OptiQ-4bit-MTPAXQ candidates: AXQ-6bit / AXQ-4bit |
| Qwen 3.8 27B | Chat / agent / multimodal | Default selector: AXQ-MXFP4-MTP via qwen3.8-27b:axq; MXFP8 MTP admission pending |
| Qwen 3.6 35B-A3B | Chat / agent / multimodal | AX-Qwen3.6-35B-A3B-MLX-4bit-MTPAX-Qwen3.6-35B-A3B-MLX-6bit-MTPAX-Qwen3.6-35B-A3B-MLX-OptiQ-4bit-MTP |
| Qwen3-VL 30B-A3B Instruct | Vision chat (image/video) | AXQ candidates: AXQ-6bit / AXQ-4bit (no MTP) |
| Holo3 35B-A3B | GUI agent (text path) | Certified AXQ: AXQ-6bit / AXQ-4bit (no MTP) |
| Ornith 1.0 35B | Coding agent | Certified AXQ: AXQ-6bit / AXQ-4bit (no MTP) |
| Tiel 35B-A3B | Coding agent | Development MXFP4 MTP: Tiel Coder / Cyber Tiel Coder; native MTP mode, no checkpoint or speed certification |
| GPT-OSS 20B / 120B | Open reasoner | Bare aliases: mlx-community MXFP4-Q4. AXQ: 20B-6bit / 120B-6bit |
| Nemotron 3 Nano 30B-A3B | Hybrid chat | AXQ candidates: AXQ-6bit / AXQ-4bit |
| Muse-Glimmer 30B | Image-text agent | AXQ candidates: AXQ-6bit / AXQ-4bit (no MTP) |
| Gemma 4 12B | Chat / multimodal | AX-Gemma-4-12B-IT-MLX-6bit-Assistant-MTPAX-Gemma-4-12B-IT-MLX-QAT-4bit-Assistant-MTPAX-Gemma-4-12B-IT-MLX-QAT-OptiQ-4bit-Assistant-MTP |
| Gemma 4 26B-A4B | Chat / agent / multimodal | AX-Gemma-4-26B-A4B-IT-MLX-6bit-Assistant-MTPAX-Gemma-4-26B-A4B-IT-MLX-OptiQ-4bit-Assistant-MTPAX-Gemma-4-26B-A4B-IT-MLX-QAT-4bit-Assistant-MTP |
| Gemma 4 31B | Chat / agent / multimodal | AX-Gemma-4-31B-IT-MLX-6bit-Assistant-MTPAX-Gemma-4-31B-IT-MLX-OptiQ-4bit-Assistant-MTPAX-Gemma-4-31B-IT-MLX-QAT-4bit-Assistant-MTP |
| Qwen3-Coder-Next | Coding agent | AX-Qwen3-Coder-Next-MLX-4bitAX-Qwen3-Coder-Next-MLX-6bit |
| DiffusionGemma 26B-A4B | Diffusion language model | AX-DiffusionGemma-26B-A4B-IT-MLX-4bit |
| EmbeddingGemma 300M | Embeddings | AX-EmbeddingGemma-300M-MLX-8bit |
| Qwen3-Embedding 0.6B | Embeddings | AX-Qwen3-Embedding-0.6B-MLX-8bit |
| Qwen3-Embedding 4B / 8B | Embeddings | AX-Qwen3-Embedding-4B-MLX-4bit-DWQAX-Qwen3-Embedding-8B-MLX-4bit-DWQ |
Native multimodal and speech support is checkpoint-authoritative: AX advertises
only modalities whose required tower tensors are present in
model-manifest.json.
| Model family | Inputs | Native API surface | Current scope |
|---|---|---|---|
| Qwen3-VL; visual Qwen3.5; Qwen 3.6 | Image, video | Chat/generate | Conv3D visual patches, full ViT/merger, MRoPE, multi-image/video ordering; Qwen 3.6 27B image/video smoke-tested on M3 Max |
| Standard Gemma 4 E4B/26B/31B | Image, video | Chat/generate | Full bidirectional ViT, 2-D RoPE, spatial pooling, checkpoint standardization, and vision projection; catalogued sizes are E4B/26B/31B. E2B still loads from an explicit directory. Conformer audio is not yet native |
| Gemma 4 unified 12B | Image, audio, video | Chat/generate | Encoder-free image/audio connector and sampled per-frame video path; requires the unified connector roles |
| MiniCPM-V 4.6 | One or more images | Chat/generate | Dynamic SigLIP grid, mid-tower merger, OCR/document prompts |
| Nemotron 3 Nano Omni | Image, audio, or both | Chat/generate | RADIO vision plus Parakeet audio with ordered mixed-media spans |
| Unlimited-OCR | Image | Native processed-input runtime; delegated OpenAI OCR profile | Full prefill KV is protected; only generated-token KV uses the decode ring |
| Unlimited-OCR-2 | Image, ordered pages | Experimental native MXFP8 SR-SWA via the Python image-request helper | Pinned-pack M5 Max functional controls; full retained KV, no constant-memory or quality qualification |
| Whisper large-v3-turbo | Audio | /v1/audio/transcriptions, /v1/audio/translations, Rust SDK |
WAV/MP3 to 16 kHz; multilingual transcribe/translate; text-generation routes fail closed |
GLM-OCR is not supported.
Download helpers inspect the source tensor index and automatically rebuild
older packaged manifests that omitted a declared Qwen or Gemma vision tower.
For an existing manual model directory, run
ax-engine-bench generate-manifest --force /path/to/model.
The default Hugging Face cache layout is
models--AutomatosX--<repository>/snapshots/<revision>. Use the shorter
ax-* aliases shown by ax-engine download --list; for example:
ax-engine serve qwen3.8-27b:axq
ax-engine serve qwen3.8-27b:axq --offline # require the pinned MXFP4 MTP cache
Aliases, hardware sizing, and legacy MTP packaging targets: Supported Models · Qwen 3.6 27B AXQ certification · Hardware FAQ · CLI.
AXQ endurance evidence
AX Engine 6.13.5 completed an 8-hour endurance test with 8.87 hours of continuous measured runtime for the pinned Qwen 3.6 27B AXQ 6-bit model on a 64 GB M4 Pro Mac mini: 437/437 requests succeeded, the owned server did not restart, and no retained-RSS growth, swap, lifecycle-drain failure, or performance guardrail breach was observed. The operator stopped this run to add deeper vLLM-style leak attribution before restarting the full 72-hour qualification, so it is short-duration evidence, not a 72-hour pass. Read the 8-hour endurance report or use the reusable AXQ endurance utility and detached launch procedure to verify another local AXQ package.
Multi-model serving
One process can keep several allowlisted models loaded and route each
request by model (OpenAI, gRPC, Ollama, Anthropic). Add mode is limited to
Qwen 3.5 9B, Qwen 3.6 27B/35B, Qwen3-Coder-Next, Gemma 4 12B/26B/31B, and the
EmbeddingGemma 300M / Qwen3-Embedding 0.6B–8B embedding models (chat +
embeddings from one process); AutomatosX AX- package names resolve to the
same targets. Each model owns its own session and scheduler while a process
arbiter fair-rotates Metal turns (no fused cross-model batch).
# After a first model is already serving on :31418
curl -s http://127.0.0.1:31418/v1/model/load -H 'content-type: application/json' -d '{
"model_id": "gemma-4-12b-it",
"model_path": "/path/to/gemma-4-12b-artifacts",
"load_mode": "add",
"make_default": false
}'
curl -s http://127.0.0.1:31418/v1/chat/completions -H 'content-type: application/json' -d '{
"model": "gemma-4-12b-it",
"messages": [{"role": "user", "content": "Hi"}],
"max_tokens": 32
}'
Full contract (load/unload, memory preflight, idle eviction, metrics labels): Server: Multi-model serving · Supported Models.
Scheduling and KV runtime
Token-level scheduling is owned by the engine, not a gateway: each step the scheduler builds a decode-first, token-budgeted batch with chunked prefill and mixed prefill+decode routes, backed by a paged logical KV ledger. Full design: Scheduler · KV Cache · Serving Invariants.
- Continuous batched decode — structurally eligible decode requests share
one batched forward (default on;
AX_MLX_BATCHED_DECODE=0is the kill switch) behind a fail-closed bit-exact certification gate; host-sampled batching is a separate opt-in - Paged KV ledger with prefix sharing — per-request block tables, ref-counted cross-request prefix reuse, tiered eviction, and an optional disk-durable prefix cache that survives restarts
- Pressure handling — KV memory-pressure throttling, preempt-and-recompute
(newest in-flight prefill only, never decode), and server admission control
(global and per-model concurrency caps →
429) - Fleet telemetry contract —
/metricspublishes versionedax_runtime_*saturation series (KV utilization, queue depth, batch headroom, TTFT p95, decode tok/s, error rate) that AX Serving and other routers consume for node selection; token scheduling itself stays on-box
Current dense batched-decode ceiling. On the strict M5 Max projection probe, the default Shared policy reaches 328.9 aggregate tok/s at B=8 (4.01× its B=1 throughput), versus 102.6 tok/s / 1.25× for the RowExact fallback. The paired Shared/RowExact ratio is 3.20× with five of five wins and identical full-cohort greedy hashes. This is a dense Llama-3.1-8B, 32-token-prefill microbenchmark—not an end-to-end serving, long-prompt, or MoE claim. Production continuous-batching claims still require matching serving evidence—see Batched decode ceiling and Long Context claim boundaries.
Performance
Two lanes. Do not mix them. Dense Qwen 3.8 27B AXQ is AX Engine's first-run pack: 20.84 GB of weights per token, already at the DRAM ceiling in direct AR. Tiel / Cyber-Tiel are AX Code's managed MoE coding packs. Their tok/s are not interchangeable with the 27B table.
| Lane | Pack | Host | AX Engine | Direct AR (mlx-lm) | MTPLX |
|---|---|---|---|---|---|
| Historical retired 6-bit (dense) | qwen3.8-27b:axq |
Mac mini M4 Pro 64 GB | 31.05 decode / 120.3 prefill | 12.78 decode (2.43×) | 28.16 / 114.0 |
| Historical retired 6-bit (dense) | qwen3.8-27b:axq |
M5 Max 128 GB | 76.90 decode / 795.3 prefill | 27.90 decode (2.76×) | 70.62 / 686.6 |
| AX Code default (MoE) | Tiel 35B MXFP4 MTP | M5 Max 128 GB | 194.88 completion (incl. TTFT) | — | 177.44 |
Decode and prefill for Qwen 27B are 20-run medians. Tiel completion includes TTFT (six-sample median). Details below; do not quote 76.90 next to 194.88 as one number.
Qwen 3.8 6-bit rows below are historical retired-pack evidence. The old AXQ selector binding selected 6-bit; it now selects MXFP4 MTP. These numbers and MTP findings do not qualify MXFP4 or MXFP8. The charts in this section measure MTP or direct decode. They do not measure DFlash2. Greedy text uses DFlash2 when its checkpoint is already present; see DFlash2.
Qualification SKU: Mac mini M4 Pro 64 GB (2026-09-17)
Dense Qwen 3.8 27B AXQ 6-bit MTP (qwen3.8-27b:axq). Direct AR already
streams 97.5% of the mini's 273 GB/s (mlx-lm 12.78 tok/s). Product-path
MTP is 31.05 tok/s here (2.43× that ceiling) and 76.90 tok/s decode /
795.3 tok/s prefill on M5 Max 128 GB (2.76× mlx-lm 27.90). That is the
dense-27B speed story: speculation past the memory wall, not a Tiel-class
token rate.
Same-pack measurements on the selected SKU (Mac16,11, macOS 26.6.2) with the
installed bundled wheel built from clean ad999f3f, which includes the
target-head and low-precision SwiGLU corrections. Same flappy contract as
below (four cases, 256 gen, greedy, thinking disabled, 2 warmups, 5 measured
reps, 3 s cooldown); decode and prefill are 20-run medians. Evidence:
benchmarks/results/mtp-axq-peer/2026-09-17-mac-mini-m4-pro-64gb/.
| Runtime | Latest checked 2026-09-17 | Decode | Prefill |
|---|---|---|---|
AX Engine 7.4.0 (product-path MTP, depth 3, build ad999f3f) |
installed wheel | 31.05 tok/s | 120.3 tok/s |
| MTPLX 2.11.3 (MTP depth 3, sustained) | PyPI Latest | 28.16 tok/s | 114.0 tok/s |
| OMLX 0.6.4 (Lightning MTP depth 1, imported sidecar) | GitHub Latest release | 15.02 tok/s | — |
| mlx-lm 0.31.3 (direct AR, no MTP) | PyPI Latest | 12.78 tok/s | — |
Three runtimes load this pack and run MTP — AX Engine and MTPLX at draft depth
3, OMLX at Lightning depth 1 — and AX Engine leads both on decode and prefill;
mlx-lm is the direct-AR (no MTP) baseline. The measured build is the 7.4.0
line (ad999f3f); the current 7.5.x releases add SSE and SDK framing fixes
that do not change throughput. The figure is generated from the campaign
summary.json by scripts/render_mtp_peer_decode_chart.py.
Two host daemons held about 1.2 CPU cores throughout; every lane ran under that condition and the AX lane was repeated with agreement within 0.4%. These are throughput numbers on the qualification SKU, not a quality or certification claim; the record stays Candidate.
Why the M4 Pro and M5 Max numbers differ: memory bandwidth
Decode on this 6-bit 27B pack streams 20.84 GB of dense weights per token. Direct autoregressive decode (mlx-lm) already uses 97.5% of the Mac mini M4 Pro's published 273 GB/s and 94.7% of the M5 Max's 614 GB/s, so the 2.2x decode gap is the 2.25x bandwidth gap. MTP moves past that ceiling by emitting about 3.9 tokens per weight pass. Prefill is compute-bound and the M5 Max has twice the GPU cores plus per-core neural accelerators, which is why its prefill lead is 6x for both AX and MTPLX. Full analysis: docs/performance/decode-bandwidth-utilization.md.
| Host | Published bandwidth | mlx-lm direct AR | Weight stream | Utilization | AX Engine MTP | Equivalent stream |
|---|---|---|---|---|---|---|
| Mac mini M4 Pro 64 GB | 273 GB/s | 12.78 tok/s | 266 GB/s | 97.5% | 31.05 tok/s | 647 GB/s (237%) |
| MacBook Pro M5 Max 128 GB | 614 GB/s | 27.90 tok/s | 581 GB/s | 94.7% | 76.90 tok/s | 1602 GB/s (261%) |
The M4 Pro MTP column is the corrected 2026-09-17 build; the M5 Max value (76.90) is the pre-correction 2026-09-15 build below, where decode moved within 0.3% after the corrections — the bandwidth ratios hold either way.
Campaign host: Apple M5 Max 128 GB (2026-09-21 refresh, AX Engine 7.5.3)
The pending refresh of the 2026-09-15 M5 Max peer table has landed: AX Engine
7.5.3 (c5d22c4a, built on the campaign host with Homebrew cargo 1.98.0 —
campaign evidence, not SKU certification) re-ran the identical contract
(flappy, four cases, 256 gen, greedy, 2 warmups, 5 measured reps, 3 s
cooldown, median of 20 runs, same snapshot directory for every runtime,
no GGUF or community-4-bit substitute) across four Qwen packs, added a
long_code suite, and measured the Gemma 4 family for the first time.
Superseded history: the 2026-09-15 7.4.0 pre-correction rows 76.90 /
795.3 (decode moved within 0.3% after the 2026-09-17 SwiGLU corrections)
stay in 2026-09-15 campaign artifacts.
Qwen (product-path MTP, mtp_head_only_verify_loop):
| Pack (suite) | AX Engine 7.5.3 | MTPLX 2.11.2 | OMLX 0.6.4 | mlx-lm 0.31.3 |
|---|---|---|---|---|
| Qwen 3.8 27B AXQ 6-bit MTP (flappy) | 76.04 / 768.7 | 73.07 / 650.6 | 37.71 | 27.92 |
| Qwen 3.8 27B AXQ 6-bit MTP (long_code) | 72.71 / 856.9 | 57.82 / 833.1 | 35.85 | 27.86 |
| Qwen 3.6 35B-A3B AXQ 6-bit MTP (flappy) | 239.52 / 2112.1 | 127.87 / 1590.7 | — | 109.97 |
| Qwen 3.8 27B AXQ MXFP4 MTP (flappy) | 76.70 / 792.4 | 63.91 / 697.3 | — | 34.20 |
Decode / prefill tok/s (prefill: AX runner-internal; MTPLX derived from
prompt_eval_time_s). mlx-lm rows are the direct-AR (no MTP) baseline.
— means the runtime could not load that pack (OMLX runs the imported 6-bit
sidecar only where one exists); an unsupported peer is never substituted
with another checkpoint.
Gemma 4 (direct AR — community checkpoints, no Assistant-MTP sidecar; the catalog-pinned AXQ Gemma 4 chat packs are not published yet):
| Checkpoint (flappy) | AX Engine 7.5.3 | MTPLX 2.11.2 | OMLX 0.6.4 | mlx-lm 0.31.3 |
|---|---|---|---|---|
gemma-4-12B-it-4bit (gemma4_unified) |
67.68 / 1626.8 | — | 60.05 | — |
gemma-4-26b-a4b-it-4bit (gemma4) |
142.48 / 2644.9 | — | 112.23 | 134.06 |
Reading this snapshot honestly: against the 2026-09-15 baseline AX moved
76.90 → 76.04 while MTPLX moved 70.62 → 73.07 at unchanged versions, so the
dense-27B short-suite margin narrowed from 1.089× to 1.041× — inside
single-host median spread, read it as parity there. The strongest signals in
this campaign are on long_code (1.26× over MTPLX) and the MoE pack
(1.87× over MTPLX at 239.5 vs 127.9 tok/s, where AX is highest in this
snapshot); the MoE margin reflects MTP-path fit on that architecture and
suite, not a standing cross-engine ranking. The Qwen 3.6 35B-A3B row is
superseded by the 2026-10-22 rerun below: MTPLX 2.12.2
reads 202.7, not 127.9, and OMLX runs on a correctly prepared pack. Gemma rows are native-graph
load/decode evidence only — direct AR, not MTP, and not the 6-bit
recommended publication lane. This is a version-pinned single-host snapshot
(2026-09-21T07:07:55Z .. 09:48:41Z UTC, sequential lanes), not a permanent
ranking, and MTP Tier 2 remains pending. Full contract, per-case medians,
limitations, and the Flash Next qwen4_exp experimental status (loads and
serves on the ADR-030 target spec; no peer version tested here loads it):
Qwen / Gemma M5 Max peer campaign.
Artifacts:
27B 6-bit ·
Qwen family ·
Gemma family.
Chart regenerated deterministically by scripts/render_m5_peer_2026_09_21.py.
Qwen 3.6 35B-A3B AXQ 6-bit MTP rerun (2026-10-22, M5 Max 128 GB)
Same pack for every runtime (AX-Qwen3.6-35B-A3B-MLX-AXQ-6bit-MTP @ 6a4c2207), the
latest release of each peer (MTPLX 2.12.2, OMLX 0.7.2, mlx-lm 0.32.2) and AX
Engine 7.8.0 (branch commit 77242419, unmerged when recorded). Contract: flappy and
long_code, 256 gen, greedy, thinking off, 2 warmups, 5 measured reps, 3 s cooldown, median
of 20 runs, CPU-idle gate before every lane, each MTP runtime at its own best draft depth.
Decode / prefill tok/s (prefill: engine-reported; OMLX reports none, its figure is prompt
tokens over client time to first token):
| Runtime | flappy decode / prefill |
long_code decode / prefill |
|---|---|---|
| AX Engine MTP | 250.6 / 2364 | 242.8 / 3550 |
MTPLX 2.12.2 (default depth 3 / best depth 2 on long_code) |
202.7 / 1765 | 247.6 / 3136 (255.5 at depth 2) |
| OMLX 0.7.2 (depth 3) | 207.1 / 1193 | 240.1 / 1963 |
| mlx-lm 0.32.2 (direct) | 113.9 / 1242 | 113.1 / 2129 |
| AX Engine direct | 122.4 / 2199 | 121.6 / 3515 |
AX decode is 1.24× MTPLX on flappy and 0.98× on long_code; AX prefill is 1.13× to
1.34× MTPLX and about 2× OMLX and mlx-lm; AX has the lowest time to first token (137 and
204 ms against 185 and 229 for MTPLX). Two findings changed this table from the
2026-09-21 one. OMLX's MTP head needs all seven RMSNorm gains shifted from raw HF deltas to
MLX multipliers; the stock AXQ pack leaves three, which gave 3.8% draft acceptance, so
OMLX now runs on the variant AXQuant's export-runtime --target omlx builds (trunk shards
byte-identical). MTPLX's lead on long_code comes from its default prompt-copy drafting
(MTPLX_CONTEXT_COPY=0 removes it: 211.1 tok/s there, 214.9 on flappy), which AX Engine
has no counterpart for on this hybrid MoE target; pure MTP head against MTPLX with copy
off is 1.17× and 1.15× in AX's favour. Single host, signal only. Details:
analysis ·
campaign record.
Qwen 3.8 Flash Next peer snapshot (2026-10-08, M2 Ultra 192 GB)
Same flappy contract as the tables above, on the pinned MXFP4 MTP pack
AutomatosX/AX-Qwen3.8-Flash-Next-MLX-AXQ-MXFP4-MTP at
ff2a28485eb89bb60e8fe35dd6c65c51e63ee7b3 (model_type=qwen4_exp).
One directory for every runtime. Host: Mac Studio, Apple M2 Ultra, 192 GiB
unified memory, internal NVMe SSD, macOS 27.0.1. AX Engine 7.6.0
(219e9c56, release binary, rustc 1.97.1, MLX 0.32.3 as recorded by the AX
harness). Lanes ran sequentially. A virtual machine was using about one full
CPU core (one-minute load average 3.8). This is a single-host snapshot. It
does not qualify MTP-S, MTP-P, or MTP-D, and it does not change the product
default. MTP Tier 2 remains pending.
| Pack (flappy) | AX Engine 7.6.0 | MTPLX 2.12.2 | OMLX 0.7.0 | mlx-lm 0.31.3 |
|---|---|---|---|---|
| Qwen 3.8 Flash Next AXQ MXFP4 MTP | 22.20 / 325.3 | — | 33.87 | — |
Decode / prefill tok/s, median of 20 measured repetitions (4 cases × 5).
Prompt lengths were 264, 273, 374, and 432 tokens. AX prefill is the
runner-internal cold-prefill figure (prefix cache disabled). OMLX does not
record a comparable prefill. — means that runtime did not serve this pack.
AX ran pure MTP (mtp_head_only_verify_loop, n-gram stacking off). The Flash
Next model policy depth on this build is 1: across the 20 repetitions it
drafted 2,550 tokens at the first MTP position and accepted 2,535 of them,
and the drafted counts at the next two positions stayed 0. Measured decode
stayed in a narrow band, 22.12–22.37 tok/s. Expert stream stayed on Auto;
the pack is not stream-required on this 192 GiB host, and the server was not
passed --stream-experts.
OMLX 0.7.0 used its VLM engine with MTP enabled and mtp_fixed_depth 3, on
MLX 0.32.2 and mlx-vlm 0.7.4. Its loader only sees embedded MTP tensors that
are listed in model.safetensors.index.json, and this pack keeps the 31
mtp.* tensors in mtp.safetensors outside that index. The lane therefore
read a same-byte symlink view whose copied index names that existing sidecar.
The pack directory was not modified. The artifact does not record how many
draft positions OMLX actually accepted, so 33.87 tok/s is not a same-width
comparison with AX depth 1. Its fused attention kernel refused to launch on
this GPU (threadgroup size 1024 exceeds the 896-thread limit) and decode
used the MLX path. The 20 measured repetitions span 31.89–54.78 tok/s: on
each case the first measured repetition was about 51–55 tok/s and the next
four were about 32–36, so the median sits in the later cluster.
MTPLX 2.12.2 rejected the pack at load: 384 PLE n-gram shard tensors are not
parameters of its qwen4_exp module. No tokens were served. mlx-lm 0.31.3
has no qwen4_exp loader, so that lane did not load weights either.
Artifacts:
2026-10-08 Flash Next peer snapshot.
Tiel / Cyber-Tiel peer refresh (2026-09-20)
These two MXFP4 MTP packs are what AX Code manages as local defaults (Tiel) and
alternate (Cyber-Tiel). They are not AX Engine's first-run Qwen 27B alias. The
four-machine performance report
reports both packs separately on M5 Max 128 GiB, M4 Pro 64 GiB, M2 Ultra
192 GiB and M3 Ultra 512 GiB. It uses identical model files and prompt tokens, fixed output counts,
cold KV, two reversed-order blocks, and completion throughput including TTFT.
These are explicit throughput-MTP, full-resident native API measurements, not
default ax-engine serve and not AX Code session speed.
Coding completion throughput includes TTFT. Each cell has six measured samples; the ranges below compare the four Python/Rust cells per host with MTPLX 2.11.3 sustained, not pooled speedups. Both engines use MLX 0.32.2.
| Tested machine | AX vs MTPLX 2.11.3 completion | Observed tradeoff |
|---|---|---|
| MacBook Pro M5 Max, 128 GiB | +3.1% to +18.0% | AX leads in completion, decode and TTFT. |
| Mac mini M4 Pro, 64 GiB | -2.4% to -0.5% | MTPLX finishes slightly faster and starts sooner; AX decode is slightly faster. |
| Mac Studio M2 Ultra, 192 GiB | -2.1% to +8.6% | Mixed completion results; AX decode is faster, MTPLX starts sooner. |
| Mac Studio M3 Ultra, 512 GiB | +2.2% to +4.7% | AX finishes faster; MTPLX starts sooner. |
This is a version-pinned snapshot against a strong, fast-moving open-source peer, not a permanent ranking: AX leads completion on M5 Max and M3 Ultra, is essentially level on M4 Pro, and is mixed on M2 Ultra. Earlier pre-residency-fix runs favored MTPLX on some hosts.
Coding completion throughput — Tiel pack (tokens/s, median incl. TTFT):
| Machine | Workload | AX Engine | MTPLX sustained | AX vs MTPLX |
|---|---|---|---|---|
| M5 Max 128 GiB | python-lru | 194.88 | 177.44 | +9.8% |
| M5 Max 128 GiB | rust-jsonl | 172.08 | 145.82 | +18.0% |
| M4 Pro 64 GiB | python-lru | 90.46 | 92.23 | -1.9% |
| M4 Pro 64 GiB | rust-jsonl | 64.62 | 64.94 | -0.5% |
| M2 Ultra 192 GiB | python-lru | 107.28 | 104.96 | +2.2% |
| M2 Ultra 192 GiB | rust-jsonl | 81.24 | 82.98 | -2.1% |
| M3 Ultra 512 GiB | python-lru | 160.35 | 153.48 | +4.5% |
| M3 Ultra 512 GiB | rust-jsonl | 127.19 | 121.80 | +4.4% |
Coding completion throughput — Cyber-Tiel pack (tokens/s, median incl. TTFT):
| Machine | Workload | AX Engine | MTPLX sustained | AX vs MTPLX |
|---|---|---|---|---|
| M5 Max 128 GiB | python-lru | 219.43 | 194.15 | +13.0% |
| M5 Max 128 GiB | rust-jsonl | 169.14 | 163.99 | +3.1% |
Decode (excludes first callback; not the primary metric):
| Machine | Pack | Workload | AX decode | MTPLX decode |
|---|---|---|---|---|
| M5 Max 128 GiB | Tiel (AX Code default) | python-lru | 217.85 | 198.40 |
| M5 Max 128 GiB | Cyber-Tiel (alternate) | python-lru | 249.01 | 219.84 |
194.88 is the public completion number for AX Code's managed default Tiel pack.
249.01 is the fastest decode cell in the campaign; it is Cyber-Tiel, not the
default, and is not an AX Code session measurement. Do not quote decode as
completion. The chart above plots both packs. AX decode is faster in every coding
cell, but the M4 Pro and M2 Ultra first-token wait offsets it on completion. The
figure is generated from the report tables by scripts/render_tiel_peer_chart.py.
Remaining Cyber-Tiel hosts and TTFT tables stay in the report.
The report includes both MTPLX profiles, separate decode/TTFT tables, memory, unwired controls and limitations. There is no comparable isolated prefill tokens/s claim. Both engines keep the model loaded throughout measured requests.
That campaign explicitly used mlx_stream_experts="off" on the 64 GiB mini;
the tested Auto build retained a 48 GiB reserve and paged experts. Wired/unwired controls and
Auto diagnostics are reported separately. These results do not promote MTP,
change defaults or certify either pack. The subsequent
bounded M4 default-session residency change
has separate AX-only server acceptance evidence; MTPLX was not rerun for that
later build, and these peer timings are not default-server measurements.
Archived Qwen 3.6 serving, multi-model S1, embeddings, and the 2026-08-31 depth-1 AX / MTPLX 2.9.0 / OMLX 0.6.4 table stay in Performance Results and Benchmarks.
SDKs
Most clients use the OpenAI-compatible HTTP server. Python also has an in-process session API.
| SDK | Docs |
|---|---|
| Rust | docs/sdk/rust.md |
| Python | docs/sdk/python.md |
| JavaScript / TypeScript | docs/sdk/javascript.md |
| Go | docs/sdk/go.md |
| Ruby | docs/sdk/ruby.md |
| Swift | docs/sdk/swift.md |
| Mojo (experimental) | docs/sdk/mojo.md |
Server
ax-engine serve is the normal entrypoint (see Quick Start). Default listen is
127.0.0.1:31418 (not AX Serving’s 18080). Port map, LAN bind, and
Serving vs Engine:
curl http://127.0.0.1:31418/v1/runtime
Auth, streaming, embeddings, Ollama-shaped routes: Server · API Compatibility · OpenClaw. Fleet / NVIDIA serving: AX Serving.
Documentation
| Need | Read |
|---|---|
| Docs hub | docs/README.md |
| Install and first request | Getting Started |
| Ports, bind host, Engine vs Serving | Ports |
| Models and MTP packages | Supported Models · MTP Docs |
| Hardware / FAQ | FAQ |
| Full performance tables | Performance Results |
| Serving peer (newest) | Serving peer detail |
| Reproduce benchmarks | Benchmarks |
| Server / API / SDKs | Server · API · OpenClaw · SDKs |
| Fleet / NVIDIA (AX Serving) | AX Serving |
| Architecture | Architecture |
| Scheduler / KV internals | Scheduler · KV Cache · Serving Invariants |
Development
cargo build --workspace
cargo test --quiet
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --check
maturin develop
python -m unittest discover -s python/tests -v
Crate layout and conventions: AGENTS.md · Architecture.
Limitations
- Qwen3.5 long-prompt prefill can trail upstream MLX references on longer prompts
- Raw HuggingFace / mlx-community snapshots load directly:
model-manifest.jsonis auto-generated fromconfig.json+ safetensors headers on first load and weight sanitization is auto-detected, somlx_lm.convertis not required - N-gram acceleration is workload-dependent, not a raw kernel speedup
- NVIDIA/CUDA serving requires the separate AX Serving product
Details: FAQ limitations.
Contributing
Issues, wishlist items, reproducible benchmarks, and docs feedback are welcome. Unsolicited code PRs for runtime, kernel, scheduler, or performance paths are generally not accepted — open an issue first. See CONTRIBUTING.md.
Community
- Website: automatosx.com
- Discord: Join us
- Email: [email protected]
Acknowledgments
AX Engine is grateful to the open-source foundations, benchmarking peers, and community contributors listed in Acknowledgments.
License
Apache License, Version 2.0. See LICENSE.
Copyright (c) 2026 DEFAI Private Limited