🎭 MoM — Mixture of Models

Transform multiple AI perspectives into one superior answer through intelligent synthesis.

MoM is a self-hosted LLM gateway that speaks the OpenAI and Anthropic wire protocols but answers each request with a panel of models instead of one. It fans a single request out to several models in parallel, then a designated concluding model (the synthesizer) reads every candidate answer and writes one consolidated response.

Think of it as assembling an expert panel: the creativity of a frontier GPT, the reasoning of Claude Opus, and the breadth of Gemini Pro — combined into a single answer that is more reliable and nuanced than any one model produces alone. Because MoM is drop-in wire-compatible, any tool that already talks to OpenAI or Anthropic — the SDKs, Claude Code, Codex, Cursor, Cline, Aider — gets ensemble answers with no code changes: just point its base URL at MoM.

🌟 Why a Mixture of Models?

In a landscape of hundreds of specialized LLMs, leaning on a single model is a self-imposed ceiling.

Each model brings its own perspective and reasoning style; MoM synthesizes them into one comprehensive answer.

Benefit	How MoM delivers it
🎯 Superior quality	Synthesis reconciles several perspectives, dropping one model's hallucinations and biases in favor of the panel's strongest reasoning.
🛡️ Resilience	If a member is slow or fails, a quorum of the others still answers — one bad seat never sinks the request.
💰 Cost control	Cheap models on the panel, a stronger one to conclude; per-tier effort, prompt caching, and relay continuations keep spend honest.
🔄 Flexibility	Hot-swap models and reshape ensembles in one YAML file — no code changes. Build task-specific "meta-models" (bmom, mom-code, …).

Real-world use cases

• 💻 Coding agents — point Claude Code / Codex / Cursor at MoM and let a panel deliberate on each step.
• 🔍 Research & analysis — consult multiple AI "experts" and get one reconciled answer.
• 📝 Content creation — combine creative and factual models for balanced, grounded writing.
• 🎓 Education — well-rounded explanations drawn from diverse reasoning styles.

🔄 How it works

A fan-out / fan-in architecture: members answer in parallel and are advisory; the synthesizer owns the client-visible output (text and tool calls).

graph LR
    A[Client request<br/>OpenAI / Anthropic wire] --> B{MoM}
    B --> M1[member A]
    B --> M2[member B]
    B --> M3[member C]
    M1 --> S{{synthesizer<br/>concluding model}}
    M2 --> S
    M3 --> S
    S --> R[one answer<br/>streamed back]

Internally there is exactly one pipeline (run_ensemble) that emits a typed event stream, and thin per-protocol encoders render that stream as Chat Completions, Responses, or Anthropic Messages. Streaming and non-streaming are two consumers of the same events, so they can never drift.

🚀 Quickstart

Requires Python 3.12+ and an API key for at least one provider.

pip install mom-llm                          # or: uv sync, to work from a checkout

mkdir -p ~/.mom                              # models and keys, once per machine
cp config.example.yaml ~/.mom/config.yaml    # or write your own
printf 'MOM_API_TOKEN=dev-secret\nOPENAI_API_KEY=sk-...\n' > ~/.mom/.env

mom config where               # what mom found, and in what order it merges
mom serve                      # http://127.0.0.1:8000

mom finds its config on a two-level search path — ~/.mom/config.yaml (or $XDG_CONFIG_HOME/mom/config.yaml) for the machine, ./mom.yaml (or ./.mom/config.yaml) for a project, deep-merged with the project winning. So a project file can be just the ensembles it adds, over the models you defined once. --config <file> or MOM_CONFIG pins one file instead.

Or with Docker — the published image, or a build from the checkout:

docker run -p 8000:8000 -e MOM_API_TOKEN=dev-secret -e MOM_CONFIG=/config.yaml \
  -v "$PWD/config.example.yaml:/config.yaml:ro" -v mom-data:/data \
  ghcr.io/arashbehmand/mom-llm:latest

docker compose up              # reads .env for secrets and MOM_CONFIG

Secrets come from the environment, or from a .env / auth.json beside any config on the search path — first definition wins, and the environment always outranks a file. The YAML config only ever names the env vars, never the values. Already authenticated with opencode? --auth-from-opencode borrows its API keys. Check the server with curl localhost:8000/health.

Coming from v1? The config format and the way you run MoM both changed (your clients don't). docs/MIGRATION.md walks through both, field by field.

🔌 API surfaces

The model you send is an ensemble from your config (e.g. bmom), not a raw provider model.

OpenAI Chat Completions — POST /v1/chat/completions

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dev-secret")
resp = client.chat.completions.create(
    model="bmom",
    messages=[{"role": "user", "content": "Compare Postgres and SQLite for a small app."}],
)
print(resp.choices[0].message.content)

Anthropic Messages / Claude Code — POST /v1/messages. Point the Anthropic SDK (or Claude Code) at MoM with two environment variables:

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=dev-secret
claude                         # Claude Code now runs against your ensemble

OpenAI Responses — POST /v1/responses (used by Codex). Any surface also works over raw HTTP:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer dev-secret" -H "Content-Type: application/json" \
  -d '{"model": "bmom", "messages": [{"role": "user", "content": "hi"}]}'

MCP — the panel as a tool call rather than a model swap, so an agent can ask for a second opinion without re-pointing its endpoint mid-session. Opt in with server.mcp: { enabled: true } (off by default) and the gateway also speaks MCP at /mcp, same port and same token:

claude mcp add --transport http mom http://localhost:8000/mcp \
  --header "Authorization: Bearer dev-secret"
mom mcp     # ...or serve the same tools over stdio, no gateway required

The tools: consult (run a configured ensemble — or a panel you assemble from the catalog for that one call — and get the synthesized answer with a per-member cost breakdown); submit, status, result and cancel, which run the same consult as a background job so a long panel never outlasts a client's tool-call timeout; answers, which returns what each member of a run said; and read-only list_llms, list_ensembles, runs, usage, and cache_stats. consult and submit take the <<SYSTEM>> directives as arguments too — exclude, only, include, synth, instruction, show_work, dedupe, cache_synth. Nothing here can purge or edit config. See docs/API.md.

Also served: GET /v1/models, /v1/models/{id}, /v1/model/info (capability cards), POST /v1/messages/count_tokens, GET /v1/metrics/usage, GET /v1/progress/{id} (SSE), and GET /health. Auth is a bearer token or an x-api-key header, compared in constant time.

✨ Features

• Three compatible surfaces — Chat Completions, Responses, and Anthropic Messages over one pipeline and a shared typed event stream, so streaming and non-streaming stay in lockstep.
• 🧰 MCP tool surface — the same pipeline exposed as tools (/mcp or mom mcp): consult a panel — waiting for it, or as a background job to poll — assemble one from the catalog on the spot, and read spend, runs, and cache state without a shell on the host. Off by default; nothing but running or cancelling a panel changes anything.
• 🖼️ Multimodal / vision — send images (OpenAI or Anthropic format); a vision request runs on the members that can see, and incapable members drop out cleanly.
• Effort tiers — a request's reasoning_effort selects a tier; each member declares its own effort per tier (a level, pass to relay the client's, off, or skip). No alias models.
• Tool calling — the synthesizer owns the tool calls; a tool result relays straight to it and skips a fresh fan-out, so multi-turn agent loops stay cheap.
• 🔎 Request-triggered web search — mark a model search-capable with a search: block; it goes online only when the client asks (see Web search below). No always-on search models.
• Honest capability cards — /v1/models reports vision, tools, reasoning, and web-search support plus a minimum context window computed from the real panel, not aspirational numbers.
• Automatic cost tracking — per-call USD from litellm's cost map for direct providers and from OpenRouter's returned usage-cost; a config pricing: block is an optional override only.
• Provider prompt caching — Anthropic cache_control breakpoints and OpenAI/xAI prompt_cache_key affinity are injected automatically.
• aiosqlite stores — a response cache (TTL + size-cap eviction, optional coalescing of identical concurrent calls) and a usage/metrics table, both on WAL SQLite with batched off-path writes.
• Survives slow turns — a client that drops a long turn no longer wastes the work: SSE keepalive heartbeats (server.stream_heartbeat) hold the connection through a slow fan-out, and fanout.detach_on_disconnect lets in-flight members finish and cache anyway, so retrying the turn hits cache and goes straight to synthesis. Default off (cancel-on-disconnect stays the safe base).

🎯 Advanced features

Thinking context — see the panel's work

Set show_work: inline on an ensemble to prepend a <think> block that shows every member's own answer before the synthesized one — useful for transparency and debugging:

<think>
Model: openai/gpt-5.6-sol
Content: [that member's answer]
---
Model: anthropic/claude-opus-4-8
Content: [that member's answer]
</think>

[the synthesized answer]

show_work: native routes it through the provider's reasoning channel instead; off (the default) hides it.

<<SYSTEM>> — steer synthesis, or reshape the panel, for one turn

(Over MCP these are ordinary tool arguments — exclude, only, include, synth, instruction, show_work, dedupe, cache_synth — so an agent needs no header block. See docs/API.md.)

Wrap directives in <<SYSTEM>>…<</SYSTEM>> in the last message of your turn. MoM strips the block from what the fan-out members see; a plain-text body becomes an instruction handed only to the concluding model, so you can steer the final synthesis without biasing the panel that feeds it:

Summarize the trade-offs of WAL mode in SQLite.
<<SYSTEM>>Answer as a terse bullet list, no preamble.<</SYSTEM>>

A few leading key: value lines are read as directives before the instruction text starts — consumed from the top for as long as the key is recognized; the first line that isn't shaped like key: value ends the directive header and everything from there on is the instruction, verbatim:

Compare these two approaches.
<<SYSTEM>>
exclude: k3, glm52
only: oai56s, cl48op
include: g31p
show_work: off
synth: cl48op
dedupe: on
Weigh whichever response cites real sources most heavily.
<</SYSTEM>>

Directive	Effect
exclude: a, b	drop these member identities from this turn's panel
only: a, b	run just these member identities (combine with exclude to remove some of those)
include: a, b	add these to this turn's panel: a member the ensemble skips at this effort tier, or any configured llm that isn't a member at all
show_work: off|inline|native	override the ensemble's configured show_work for this turn
synth: llm-name	run synthesis on a different configured llm for this turn
dedupe: on|off	override server.dedupe for this turn: on attaches an identical concurrent turn to the run already in flight, off forces a fresh one
cache_synth: on|off	override cache.synthesis for this turn: on keeps the synthesized answer for the next identical turn (and lets it finish in the background if you disconnect), off forces a fresh one

Identities are the as:/llm names shown in the think block and the progress dashboard; include: also takes any llm name from the catalog, which joins the panel under its own name running its own configured params. The three are applied in order — only:, then exclude:, then include: — so include: wins over exclude: on the same name, and only: fast + include: k3 builds a two-model panel from scratch.

A directive MoM can't honor doesn't cost you the turn. An unknown member name, an unknown synth: target, a value outside a directive's vocabulary, a mistyped key: each is ignored and reported at the top of the think block — on every surface and whatever show_work says, because a panel that quietly ran the wrong roster is the failure this exists to prevent. Near misses name the likely culprit ('k33' is not a member of ensemble 'emom' — ignored. Did you mean 'k3'?), and every notice is logged. The one thing still fatal is a selection with nothing left to run — an only: that matches nothing, or an exclude: that drops the panel below the ensemble's quorum — which stays a clean 400 before any fan-out spend, since there is no answer to be had either way.

Blank lines inside the header are ignored, so the block works typed into a markdown chat box — which puts one after the opening tag and between every line you press Enter on. An unknown key ends the directive header instead of being swallowed: that line and everything after it become the instruction, verbatim, with a warning naming the key — nothing you typed is lost. If your instruction text genuinely needs to start with something shaped like Word: …, prefix it with the instruction: directive, which ends the header explicitly and skips the warning.

For the same shaping made permanent — this machine always runs emom without a particular model — use members_exclude / members_include in a config layer instead of retyping a directive every turn.

The older <<CONCLUDING-INSTRUCTION>>…<</CONCLUDING-INSTRUCTION>> marker still works exactly as before (instruction-only, no directive header) — <<SYSTEM>> is just the generalized form.

Web search

A model becomes search-capable by carrying a search: block; its provider search params are merged in only when the client requests web search (web_search / web_search_options, an Anthropic web_search tool, or a Responses web_search tool). Offline requests leave it untouched.

llms:
  k3:  # OpenRouter: the web plugin, request-triggered (no separate always-on ":online" model)
    model: openrouter/moonshotai/kimi-k3
    search: { extra_body: { plugins: [{ id: web }] } }
  gemini:  # Google Search grounding
    model: gemini/gemini-3.1-pro
    search: { web_search_options: { search_context_size: high } }

⚙️ Configuration

One YAML file, version: 2, with three name-keyed maps: llms: (individual models, with extends inheritance), prompts: (synthesis instructions), and ensembles: (what clients call). Each ensemble lists advisory members, a synthesizer, a strategy (synthesize or passthrough), and per-member effort. Secrets never appear — only env-var names.

Config comes from a search path, deep-merged like git config: a user level (~/.mom/config.yaml) for the models a machine has once, a project level (./mom.yaml) for what a directory adds, and a gitignored *.override.yaml beside each for what belongs to this machine alone. A later layer masks an inherited key with null — and shapes a panel it did not author with members_exclude / members_include, since a members: list would otherwise have to be restated in full to drop one model from it:

# ~/.mom/config.override.yaml — untracked, this machine only
ensembles:
  emom:
    members_exclude: [fable, astra]    # the tracked roster keeps them; this machine doesn't run them

Start from config.example.yaml and see docs/CONFIGURATION.md for the full reference. Validate and inspect with:

mom config where           # every path checked, what was found, and the merge order
mom config validate        # loads, validates, resolves; non-zero on any problem
mom config show bmom       # flattened, resolved view of one ensemble

Each of those takes an optional path (mom config show config.example.yaml bmom) when you want to inspect a specific file rather than whatever is on the search path.

🔗 Use MoM as a provider

MoM is a drop-in base_url for the OpenAI SDK, the Anthropic SDK and Claude Code (ANTHROPIC_BASE_URL), Codex (Responses API), and Cursor / Cline / Aider. See docs/PROVIDERS.md for copy-paste per-tool setup.

📚 Documentation

• docs/ARCHITECTURE.md — the pipeline, event stream, and hexagonal layering
• docs/API.md — endpoint and wire-format reference
• docs/CONFIGURATION.md — the v2 config schema
• docs/PROVIDERS.md — using MoM from SDKs and agent tools
• docs/MIGRATION.md — upgrading from v1 (deployment + config)
• CHANGELOG.md — what landed in each release

🛠️ Development

uv sync                 # dev + test dependency groups
uv run pytest           # incl. SDK-in-the-loop contract tests

Quality gates: ruff, mypy --strict, and import-linter, which enforces the layered architecture and keeps litellm quarantined in a single adapter module.

License

MIT — see LICENSE.