Lattice

Pure Rust inference engine for transformer models on Apple Silicon, with a native macOS app.

Quick start · Lattice Studio · Benchmarks · Roadmap

No ONNX. No Python. No CUDA. No external ML runtime. Lattice implements the full compute graph in Rust: weight loading, tokenization, forward pass, vector operations, quantization, and LoRA training. Hand-written Metal shaders accelerate inference on Apple Silicon. SIMD kernels (AVX2 on x86, NEON on ARM) handle the CPU path.

---

What is Lattice

Lattice is two things in one repo.

A Rust inference library. A workspace of focused crates covering embeddings, generation, quantization, LoRA fine-tuning, and optimal transport. The canonical member list is in Cargo.toml. Use lattice-embed as a library dependency, or run the lattice binary for interactive chat and an OpenAI-compatible HTTP server.

Lattice Studio: a native macOS app. A SwiftUI instrument panel that drives the Rust engine via CLI subprocesses. Train LoRA adapters with a live loss oscilloscope, quantize models with a before/after comparison, run multi-turn chat with persistent GPU model residency, and manage your model library from a single window. To build and install it, follow the step-by-step guide in apps/macos/INSTALL.md.

---

Capabilities

Pure Rust compute	Hand-written SIMD kernels (AVX2/NEON). No C++, no ONNX, no CUDA.
Metal GPU backend	Native Apple Silicon acceleration via Metal MSL shaders. WGPU fallback for cross-platform.
Generation models	Qwen3.5-0.8B / 2B via lattice chat/serve. Qwen3.6-27B via lattice chat/serve from a native Q4 checkpoint (requires the Metal GPU build, --features "f16 metal-gpu"); safetensors 27B is loader-level only. Qwen3.6-35B-A3B (MoE): config + weight loader support. Hybrid GatedDeltaNet + GQA architecture.
Embedding models	9 models: BGE, E5, MiniLM, Qwen3-Embedding families. Auto-download for 7 BERT-family variants.
Tokenizers	WordPiece, SentencePiece, Qwen-style BPE, and Gemma BPE. No Hugging Face tokenizers C extension.
Quantization	Q8, Q4, and QuaRot (rotation-based 4-bit), including Q4 + LoRA inference on Qwen3.5.
LoRA	Inference hooks, load-time adapter selection, PEFT safetensors format, training via lattice-tune.
HTTP API	OpenAI-compatible /v1/chat/completions via lattice serve.
Safetensors native	Memory-mapped weight loading. Single-file and sharded checkpoints.
KV cache	Incremental decoding with key-value caching.
Speculative decoding	Draft-model acceleration on the CPU path.
Grammar decoding	Constrained output via a pushdown automaton. OpenAI string-level stop sequences.
MRL support	Matryoshka truncation for Qwen3-Embedding models (output dimension >= 32).
LRU cache	CachedEmbeddingService with sharded in-memory cache and hit/miss stats.
Knowledge distillation	Distillation pipeline in lattice-tune: teacher-label schema, batching, and stats. Live Claude/GPT/Gemini teacher transport is not yet wired; labeling fails closed rather than simulating (a deterministic simulated-teacher path exists behind a non-default feature for tests).
Optimal transport	Sinkhorn-Knopp solver for embedding drift detection via lattice-transport.

---

Benchmarks

The context-scaling benchmark estimates incremental generation throughput by comparing requested output-token budgets with an 8-token baseline. For each budget N (64, 128 and 256 by default), it reports (N - 8) / (T(N) - T(8)) tokens/s, where T(N) is the median measured duration in seconds. The Context labels identify requested output-token budgets.

Lattice times chat_completion() after model loading, including prompt processing, prefill and generation. Subtracting the baseline is intended to cancel common prefill and fixed per-call costs. The MLX leg receives an explicit 4-token warmup. The context-scaling Ollama leg uses the API's eval_duration, which records output-token generation separately from model loading and prompt evaluation. This excludes those durations even in a single-run invocation. Older context-scaling reports used Ollama's total_duration; their Ollama timings and derived slopes are not directly comparable with the corrected metric. Regenerate the measurements before making that comparison (#1060). These estimates do not measure end-to-end interactive latency, which also depends on model loading, prompt length and other request costs (see "Why throughput falls with context" below).

The cross-engine decode table previously published here has been withdrawn. An audit traced its Lattice and Ollama columns to an arithmetic bug in a retired reporting script: the slope computation truncated an intermediate division to one decimal place, inflating those results by up to 19% (#1060). That shell pipeline has since been replaced by scripts/bench_decode_harness.py (#938). The raw timings in docs/bench_results/context_scaling.tsv were unaffected by the bug and its slope column has been recomputed exactly from them — but the underlying measurements are still a single-session snapshot on one machine, so the cross-engine table stays withdrawn until a multi-session baseline program (medians with disclosed variance, generated by scripts/bench_decode_adapters_context_scaling.py) replaces it. Raw measurements remain in docs/bench_results/ with provenance annotations; reproduce with uv run --with mlx-lm python3 scripts/bench_decode_harness.py run --profile context_scaling (see "Context scaling" under Development below for the full invocation).

Why throughput falls with context

Decode slows as context grows because every generated token attends over the entire KV cache:

• The grouped-query attention (GQA) layers do work proportional to context length on each token: the full key/value cache is streamed through memory every step. Decode is memory-bandwidth bound, so per-token latency rises linearly with context (t ≈ intercept + slope * ctx) and throughput is its reciprocal.
• The linear-attention (gated-delta) layers are constant-time in context and do not contribute to the slope.

In an interactive chat this compounds with prompt prefill, a separate and larger cost than the decode slope and historically the main reason a long chat felt progressively slower. Both serve binaries now reuse the previous turn's shared token prefix instead of re-prefilling the whole conversation from scratch on every turn (#462, see docs/cross-turn-cache.md); a request whose history is not a safe append onto the retained state still falls back to a full re-prefill.

MLX uses Apple's private MPS/AMX matrix engines. Lattice uses the public Metal compute API, the same tier as Ollama. Lattice's edge is portability (pure Rust, zero Python, zero framework) plus capabilities neither Ollama nor MLX provide for this model family:

Capability	Lattice	MLX	Ollama
QuaRot 4-bit (rotation-based quant)	yes	no	no
Q4 + LoRA inference	yes	no	no
Pure Rust, zero Python or framework	yes	no	no

PPL benchmark (wikitext-2, from docs/bench_results/perplexity.tsv, regenerated 2026-07-07): Lattice q4 16.59, q4-QuaRot 19.01 (2048 tokens); MLX q8 15.82, q4 18.18 (2041 tokens, carried from a prior reference run — see the TSV's provenance comments). Reproduce: ./scripts/bench_quality.sh

---

Crates

Crate	Description	crates.io
lattice-embed	Embedding service: EmbeddingService trait, NativeEmbeddingService, CachedEmbeddingService, SIMD cosine/dot/euclidean, backfill
lattice-inference	Transformer kernel: safetensors loading, BERT/BGE/Qwen3 forward pass, tokenizer facade, Metal/WGPU backends, LoRA hooks, KV cache, speculative decoding, quantization
lattice-fann	Fast neural network primitives: NetworkBuilder, pre-allocated layers, zero-alloc forward pass, backprop trainer
lattice-tune	Training: knowledge distillation pipeline, dataset management, LoRA adapter management, model registry
lattice-tokenizer	Pure Rust tokenizers: WordPiece, SentencePiece, Qwen and Gemma BPE, model-directory auto-detection; re-exported by lattice-inference
lattice-grammar	Grammar-constrained decoding: JSON Schema and GBNF compilation, byte-level PDA, vocabulary masks; re-exported by lattice-inference
lattice-weights	Weight storage: safetensors, f32/f16/Q3/Q4/Q8 representations and model-directory format detection; re-exported by lattice-inference
lattice-quant	Quantization primitives: bounded Q4 manifest reading, Hadamard transforms and rotation absorption; re-exported by lattice-inference
lattice-arch	Model interface: generation types and sampling; validated config.json parsing for Qwen3.5, Gemma 4, EmbeddingGemma 2 and BitNet; re-exported by lattice-inference
lattice-compute	Host-side compute: RoPE tables, KV caches, prefix reuse and LoRA hooks; re-exported by lattice-inference
lattice-model-io	Model files: cache directory, checksum-verified downloads and environment switches; re-exported by lattice-inference
lattice-inference-core	Shared base types: InferenceError, re-exported by lattice-inference
lattice-metal-runtime	Shared Metal buffers, command encoding and pipeline construction; macOS with the optional metal-gpu feature
lattice-serve	HTTP server and the lattice CLI: OpenAI-compatible routes, model workers, LoRA serving; installed with cargo install lattice-serve
lattice-transport	Optimal transport: Sinkhorn-Knopp (balanced + unbalanced), Wasserstein barycenters, embedding drift detection

Three leaf crates (fann, transport, inference-core) have zero intra-workspace dependencies. tokenizer, grammar and weights each depend only on inference-core at runtime, as does model-io. arch depends on inference-core and weights; quant depends on those two and on arch, and compute on inference-core, arch, tokenizer and weights. Those four compute dependencies are unconditional; its external gemm dependency is enabled only by the opt-in x86-gemm feature. inference depends on all eight unconditionally and pulls in fann through the optional mixture or metal-gpu features. Each can be used on its own.

---

Quick Start: Embeddings

[dependencies]
lattice-embed = "0.12"
tokio = { version = "1", features = ["full"] }

use lattice_embed::{EmbeddingService, EmbeddingModel, NativeEmbeddingService};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let service = NativeEmbeddingService::default();

    // Single embedding (BGE-small-en-v1.5, 384 dimensions)
    let embedding = service
        .embed_one("The quick brown fox jumps over the lazy dog", EmbeddingModel::default())
        .await?;

    println!("Dimensions: {}", embedding.len()); // 384

    // Batch
    let texts = vec![
        "First document".to_string(),
        "Second document".to_string(),
    ];
    let embeddings = service.embed(&texts, EmbeddingModel::BgeSmallEnV15).await?;

    // SIMD-accelerated similarity
    let similarity = lattice_embed::utils::cosine_similarity(&embeddings[0], &embeddings[1]);
    println!("Similarity: {:.4}", similarity);

    Ok(())
}

Model weights are downloaded from HuggingFace on first use and cached at ~/.lattice/models (or $LATTICE_MODEL_CACHE).

GPU acceleration (macOS)

lattice-embed = { version = "0.12", features = ["metal-gpu"] }

lattice-embed does not have a wgpu-gpu feature; cross-platform WGPU acceleration is a lattice-inference-level feature (see the Feature Flags table below) and is not currently forwarded through lattice-embed.

---

Quick Start: CLI

Install

Three ways to get lattice, in order of convenience:

1. cargo install (from crates.io):

# CPU build (Linux/macOS). BF16/F16 safetensors load in every feature set;
# the f16 feature adds F8_E4M3/F8_E5M2 tensor loading and no dependencies.
cargo install lattice-serve --bin lattice --features f16

# With Metal GPU (macOS only)
cargo install lattice-serve --bin lattice --features f16,metal-gpu

This installs lattice to ~/.cargo/bin/lattice. Requires Rust 1.93+ (rustup update if cargo install complains about the rust-version). Releases before 0.16 shipped the CLI in lattice-inference (cargo install lattice-inference --bin lattice).

2. Prebuilt release binaries:

GitHub releases ship lattice-<version>-<target>.tar.gz for aarch64-apple-darwin (macOS, Metal-enabled), x86_64-unknown-linux-gnu, and aarch64-unknown-linux-gnu (both CPU-only), plus a matching .sha256 file. Check the release's Assets list first: releases published before this workflow landed have no prebuilt binaries (use cargo install for those versions).

VERSION=<version>   # replace with a release whose Assets list includes lattice-* tarballs
TARGET=aarch64-apple-darwin   # or x86_64-unknown-linux-gnu / aarch64-unknown-linux-gnu

curl -LO "https://github.com/ohdearquant/lattice/releases/download/v${VERSION}/lattice-${VERSION}-${TARGET}.tar.gz"
curl -LO "https://github.com/ohdearquant/lattice/releases/download/v${VERSION}/lattice-${VERSION}-${TARGET}.tar.gz.sha256"

# Verify before extracting
shasum -a 256 -c "lattice-${VERSION}-${TARGET}.tar.gz.sha256"

tar -xzf "lattice-${VERSION}-${TARGET}.tar.gz"
./lattice-${VERSION}-${TARGET}/lattice chat --model ~/.lattice/models/qwen3.5-0.8b

No Homebrew tap yet — tracked as a follow-up once release binaries have shipped for a few versions (#633).

3. Build from source (requires Rust 1.93+ and, for Metal, macOS 14+):

git clone https://github.com/ohdearquant/lattice
cd lattice

# CLI binary (chat + serve). BF16/F16 safetensors load in every feature
# set; the f16 feature adds F8_E4M3/F8_E5M2 tensor loading and no dependencies.
cargo build --release -p lattice-serve --bin lattice --features f16

# Interactive chat
./target/release/lattice chat --model ~/.lattice/models/qwen3.5-0.8b

# Generate one response and exit
./target/release/lattice chat --model ~/.lattice/models/qwen3.5-0.8b -p "Hello"

# OpenAI-compatible HTTP server
./target/release/lattice serve --model ~/.lattice/models/qwen3.5-0.8b --port 8080

chat -p (or --prompt) uses the supplied text literally, including whitespace and newlines, and never reads stdin. It writes the response to stdout; loading messages and errors go to stderr. Generation or output errors return a nonzero exit status. Omit the flag for the interactive loop.

With Metal GPU (macOS only):

cargo build --release -p lattice-serve --bin lattice --features metal-gpu,f16

Get a model

The embedding models above download automatically on first use. Generation checkpoints for lattice chat and lattice serve are not auto-fetched yet, so download one into the model cache before your first chat:

pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3.5-0.8B --local-dir ~/.lattice/models/qwen3.5-0.8b

huggingface_hub is only the download transport — it is not a runtime dependency, and the engine itself runs no Python. Then point --model at that directory:

lattice chat --model ~/.lattice/models/qwen3.5-0.8b

Beyond the unified lattice binary, several standalone tools live in crates/inference/src/bin/ for quantization, direct Metal-GPU chat, perplexity scoring, and LoRA-mixture benchmarking — see docs/cli-tools.md for verified command sequences and flag references.

Raspberry Pi / ARM Linux (aarch64)

Lattice builds and runs on 64-bit ARM Linux with no GPU: the CPU path dispatches hand-written NEON kernels at runtime. Linux aarch64 is part of CI (ubuntu-24.04-arm), and the same steps apply to a Raspberry Pi running a 64-bit OS.

Requirements and tips:

• 64-bit OS required. Raspberry Pi OS (64-bit) or Ubuntu Server for Pi. Check with uname -m — it must say aarch64, not armv7l.

• Rust via rustup (1.93+): curl https://sh.rustup.rs -sSf | sh, then build with --features f16 exactly as above — without it, BF16/F16 checkpoints fail at load time. Leave metal-gpu off — it is macOS-only.

• Memory: 8 GB strongly recommended for chat models. The CPU path converts bf16 weights to f32 in RAM, so Qwen3.5-0.8B occupies roughly 3.2 GB steady-state plus KV cache and activations — and the load-time peak is higher still, because the mapped bf16 source and converted f32 tensors coexist during conversion. On a 4 GB board this will swap-thrash or be OOM-killed. Embedding models are much smaller (bge-small is ~130 MB) and run comfortably on 4 GB.

• Get a chat model the same way as on macOS — download from HuggingFace and point --model at the directory:

  pip install -U "huggingface_hub[cli]"
  huggingface-cli download Qwen/Qwen3.5-0.8B --local-dir ~/.lattice/models/qwen3.5-0.8b
  ./target/release/lattice chat --model ~/.lattice/models/qwen3.5-0.8b
  

• Compile time: a release build takes a while on a Pi (expect tens of minutes on a Pi 5). Cross-compiling from a faster machine with target aarch64-unknown-linux-gnu works too.

• Set expectations: this is a CPU-only, memory-bandwidth-bound path. Decode speed is far below the Apple Silicon numbers in the benchmark table above — treat a Pi as a functional target for experimenting with a fully self-contained Rust inference stack, not a fast one. We haven't published Pi throughput numbers yet; benchmark reports from real boards are welcome in the issues.

HTTP API

lattice serve exposes an OpenAI-compatible endpoint:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-0.8b",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

---

Lattice Studio (macOS App)

Lattice Studio is a native SwiftUI app for macOS 14+. It wraps the Rust engine in an instrument-panel interface: live loss curves, before/after quantization comparisons, multi-turn chat, and a model library manager.

New to the app? apps/macos/INSTALL.md is a step-by-step install, get-a-model, and first-chat guide. The reference below covers building the bundle and what each screen does.

Build and package

# Requires: Xcode (Swift 6.3+), Rust toolchain
./apps/macos/scripts/package-app.sh

This builds the Swift frontend, compiles all ten Rust engine binaries in release mode, and assembles a self-contained Lattice.app bundle at apps/macos/dist/. Lattice.dmg and Lattice.zip are also produced. The packaged app needs no Rust toolchain on the recipient machine.

# Skip Swift rebuild (use existing build output)
./apps/macos/scripts/package-app.sh --skip-build

# Skip Cargo rebuild
./apps/macos/scripts/package-app.sh --skip-cargo

Install

Drag Lattice.app from the .dmg to /Applications.

The app is ad-hoc signed. On first launch, right-click and choose "Open" to bypass Gatekeeper, then click "Open" in the dialog. macOS remembers the exception for subsequent launches. Alternatively:

xattr -dr com.apple.quarantine /Applications/Lattice.app

What is in Lattice Studio

Models (cmd-1). A table of all local models and adapters under ~/.lattice/models, with file manifests, config details (18 GatedDeltaNet + 6 GQA layers called out explicitly), and Download/Verify/Reveal in Finder actions.

Train (cmd-2). LoRA fine-tuning with a live loss oscilloscope. A 56-point hero loss numeral ticks digit-by-digit as the run progresses. Scrub the loss curve to freeze all metric readouts (lr, grad-norm, tok/s, ETA) to any step. Results stream as line-delimited JSON from the engine.

Quantize (cmd-3). Q4 or QuaRot quantization. A side-by-side comparison shows size, bits, and estimated PPL before and after. True-scale bars animate to show the compression ratio. A gate pill states the result: PASS, WARN, or FAIL.

Chat (cmd-4). A single streaming transcript replays completed turns into each prompt. The GPU path reuses a persistent chat_metal --serve process so the selected model stays resident between turns; the CPU path launches generate_lora once per turn. The shipped Chat screen does not yet expose adapter selection or parallel base-vs.-adapter comparison.

Data (cmd-5). Paste raw text or load files. Preview the derived {prompt, completion} pairs as a JSONL table, validate token counts (using the same tokenizer the engine uses), and export straight into the Train dataset field.

Runs (cmd-6). Archive of every training and quantization run. Select a row to reopen its exact config and view the frozen loss curve.

The command bar (cmd-K) runs everything from a single keyboard shortcut: train qwen3.5 r8 or quantize quarot parse into argument chips and fire a run.

---

Supported Models

Embedding models

Variant	HuggingFace ID	Dims	Max tokens	Auto-download
BgeSmallEnV15	BAAI/bge-small-en-v1.5	384	512	yes
BgeBaseEnV15	BAAI/bge-base-en-v1.5	768	512	yes
BgeLargeEnV15	BAAI/bge-large-en-v1.5	1024	512	yes
MultilingualE5Small	intfloat/multilingual-e5-small	384	512	yes
MultilingualE5Base	intfloat/multilingual-e5-base	768	512	yes
AllMiniLmL6V2	sentence-transformers/all-MiniLM-L6-v2	384	256	yes
ParaphraseMultilingualMiniLmL12V2	sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2	384	128	yes
Qwen3Embedding0_6B	Qwen/Qwen3-Embedding-0.6B	1024	2048	local dir only
Qwen3Embedding4B	Qwen/Qwen3-Embedding-4B	2560*	2048	local dir only

*Qwen3-Embedding-4B and 0.6B support MRL truncation to any dimension >= 32. BGE v1.5 uses CLS pooling. E5 and MiniLM use mean pooling. E5 embed_passage() applies the "passage: " prefix automatically.

Generation models (local files required)

Config preset	Description
Qwen35Config::qwen35_0_8b	24 layers, 1024 hidden, 1 MTP layer. Base decode shipped; MTP experimental.
Qwen35Config::qwen35_2b	24 layers, 2048 hidden, dense FFN, tied embeddings.
Qwen35Config::qwen36_35b_a3b	40 layers, MoE 256 experts top-8. Config and weight loader supported.
Qwen35Config::qwen36_27b	64 layers, 5120 hidden, dense FFN. Runs in lattice chat/serve from a native Q4 checkpoint (Metal GPU build).

The Qwen3.5 architecture uses a hybrid of 18 GatedDeltaNet layers and 6 GQA attention layers. Lattice is the only open-source engine that correctly runs this hybrid recurrence at Q4 on Apple Silicon.

---

Model Selection (Embeddings)

use lattice_embed::EmbeddingModel;

// Fast English retrieval
let model = EmbeddingModel::BgeSmallEnV15;   // 384-dim, fastest, auto-download

// Balanced quality/speed
let model = EmbeddingModel::BgeBaseEnV15;    // 768-dim, auto-download

// Best quality, English
let model = EmbeddingModel::BgeLargeEnV15;   // 1024-dim, auto-download

// Multilingual retrieval
let model = EmbeddingModel::MultilingualE5Base;  // 768-dim, 100+ languages

// Decoder-based, multilingual (local files required)
let model = EmbeddingModel::Qwen3Embedding0_6B;  // 1024-dim, 2K context

// MRL: variable output dimension
use lattice_embed::ModelConfig;
let config = ModelConfig::try_new(EmbeddingModel::Qwen3Embedding4B, Some(512))?;

---

Vector Operations

lattice-embed exposes SIMD-accelerated vector utilities as a stable public API:

use lattice_embed::utils;

// Runtime dispatch: AVX2 on x86_64, NEON on aarch64, scalar fallback elsewhere
let sim = utils::cosine_similarity(&a, &b);
let dot = utils::dot_product(&a, &b);
let dist = utils::euclidean_distance(&a, &b);

utils::normalize(&mut vector);  // in-place L2 normalization

// Batch operations
let sims = utils::batch_cosine_similarity(&pairs);

Measured performance on normalized vectors (internal benchmarks, subject to hardware):

Operation	Scalar	SIMD
cosine similarity (384-dim)	~650 ns	~90 ns
cosine similarity (768-dim)	~1300 ns	~180 ns
cosine similarity (1024-dim)	~1700 ns	~240 ns

---

Feature Flags

lattice-embed

Feature	Default	Description
native	yes	Pure Rust inference via lattice-inference
image	yes	PNG/JPEG decoding for vision embeddings (implies native)
download	yes	First-use model downloads (implies native)
metal-gpu	no	Metal GPU acceleration (macOS)
avx512	no	Deprecated no-op — the AVX-512 VNNI int8 kernel is compiled unconditionally on stable x86_64 and is selected at runtime only when AVX-512F/BW/VNNI are all detected; otherwise AVX2 or scalar runs

lattice-inference

Feature	Default	Description
f16	no	Half-precision weights
metal-gpu	no	Metal compute backend
wgpu-gpu	no	WGPU cross-platform GPU backend
download	yes	HuggingFace weight download with checksum verification
image	yes	PNG/JPEG decoding; also enabled by lattice-serve
backfill	no	Re-embedding coordinator (requires rusqlite)

---

Architecture

flowchart TD
    App[Application] --> Embed[lattice-embed]
    Embed -.->|native, enabled by default| Inference[lattice-inference]
    Inference --> Tokenizer[lattice-tokenizer]
    Inference --> Core[lattice-inference-core]
    Tokenizer --> Core
    Inference --> Grammar[lattice-grammar]
    Inference --> Weights[lattice-weights]
    Inference --> Quant[lattice-quant]
    Inference --> Arch[lattice-arch]
    Inference --> Compute[lattice-compute]
    Inference --> ModelIo[lattice-model-io]
    ModelIo --> Core
    Compute --> Core
    Compute --> Arch
    Compute --> Tokenizer
    Compute --> Weights
    Grammar --> Core
    Weights --> Core
    Quant --> Core
    Quant --> Weights
    Quant --> Arch
    Arch --> Core
    Arch --> Grammar
    Arch --> Weights
    Inference -.->|mixture or metal-gpu| Fann[lattice-fann]
    Inference -.->|metal-gpu| MetalRuntime[lattice-metal-runtime]
    MetalRuntime --> Core
    Tune[lattice-tune] --> Fann
    Tune -.->|inference-hook or train-backward| Inference
    Serve[lattice-serve] --> Inference
    Transport[lattice-transport]

Arrows point from consumers to dependencies; dashed arrows are feature-gated. The diagram shows production dependencies. lattice-tokenizer owns tokenization, with supported compatibility exports under lattice-inference::tokenizer. lattice-inference-core owns shared base types such as InferenceError.

Inference uses CPU kernels (AVX2/NEON) by default. Metal on macOS (metal-gpu) and WGPU for cross-platform acceleration (wgpu-gpu) are opt-in backends. lattice-tune builds without inference by default; its inference integration requires inference-hook or train-backward.

---

Running Benchmarks

Embedding throughput

cargo bench --package lattice-embed

Metal GPU decode (macOS only, requires model weights)

cargo bench -p lattice-inference --features metal-gpu,f16 -- metal_decode

Context scaling (Qwen3.5-0.8B)

uv run --with mlx-lm python3 scripts/bench_decode_harness.py run --profile context_scaling --allow-missing-engine

Performance depends on hardware, model size, batch size, and sequence length. Run benchmarks on your target hardware for representative numbers.

---

Roadmap

Three capability lanes are next for the engine:

• Vision tensors: image input through the pure-Rust graph, from patch embedding through a vision encoder and projector into the language model.
• Audio tensors: on-device speech, starting with a mel-spectrogram frontend and audio encoder for speech input; speech output follows.
• Gemma 4 model family: broaden the engine beyond Qwen with a second model family.

Tracking issues with first-milestone slices: see the issue tracker.

---

Documentation

• Architecture: crate dependency graph, design decisions, stability tiers
• Models: full model support matrix, attention variants, inference features
• Getting started: step-by-step setup guide
• Examples: code samples for common tasks
• CLI tool walkthroughs: verified command sequences for quantize_q4, chat_metal's full flag surface, ppl_metal, and bench_lora_mixture
• Capability matrix: lattice CLI vs lattice_serve endpoint/field parity
• ADR directory: docs/adr/

---

Development checks

make lint-docs checks tracked Markdown with Deno and runs the capability-fixture, path and source-marker checks. The source-marker check requires ripgrep (rg) and fails rather than skipping when it is absent, so it never reports a tree it did not scan. Without Deno, the default command still runs the other checks and names the skipped Markdown checks in its final line. --format and --markdown-only always require Deno.

The CI job that installs Deno requires its presence. Other verification machines provisioned with Deno should opt in too:

LATTICE_REQUIRE_DENO=1 make lint-docs

This opt-in exits 127 when Deno is unavailable. Leaving it unset or setting it to 0 preserves the default behavior for contributors without Deno. Export the variable when the same requirement should apply to make ci or Git hooks.

Publishing

scripts/publish.sh publishes dependencies before dependents:

1. fann, transport, and inference-core (leaves).
2. tokenizer, grammar, weights, metal-runtime, and model-io.
3. arch.
4. quant and compute.
5. The model-family crates under crates/models/.
6. inference.
7. embed, tune and serve.

The script waits 30 seconds between tiers for registry indexing. Its dry-run mode checks package sizes for the workspace crates, then runs cargo publish --dry-run for the leaf tier only; later tiers require this release's dependencies on crates.io.

make publish

---

License

Licensed under either of MIT or Apache-2.0, at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work shall be dual-licensed as above, without any additional terms or conditions.

Built by Ocean (HaiyangLi). Powers khive, a cognitive infrastructure for AI agents.