MYRTIA

Agent-native red teaming for large language models and AI agents.

myrtia.ai

A council of attacker models deliberates, votes and strikes — and you watch it live.

▶ Watch the demo video

Quick Start

Myrtia installs with uv: clone, then uv run myrtia. uv creates the environment, downloads the right Python (3.12 — no need to upgrade your system Python) and installs everything — all 21 attacks, not just the council. The default models run locally in Ollama, no API key needed.

Copy-paste the block for your OS into a terminal. It installs uv and Ollama, waits for Ollama to start, pulls the demo models (qwen2.5:7b attacks and judges, gemma3:1b is the target), then clones Myrtia and opens it at http://127.0.0.1:8000/.

curl -LsSf https://astral.sh/uv/install.sh | sh && source "$HOME/.local/bin/env"
brew install ollama && brew services start ollama
until curl -s http://127.0.0.1:11434 >/dev/null; do sleep 1; done
ollama pull qwen2.5:7b && ollama pull gemma3:1b
git clone https://github.com/francordel/myrtia.git && cd myrtia
uv run myrtia

curl -LsSf https://astral.sh/uv/install.sh | sh && source "$HOME/.local/bin/env"
curl -fsSL https://ollama.com/install.sh | sh
until curl -s http://127.0.0.1:11434 >/dev/null; do sleep 1; done
ollama pull qwen2.5:7b && ollama pull gemma3:1b
git clone https://github.com/francordel/myrtia.git && cd myrtia
uv run myrtia

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
winget install Ollama.Ollama

ollama pull qwen2.5:7b && ollama pull gemma3:1b
git clone https://github.com/francordel/myrtia.git; cd myrtia
uv run myrtia

curl -LO https://raw.githubusercontent.com/francordel/myrtia/main/compose.yaml
docker compose up

curl -LO https://raw.githubusercontent.com/francordel/myrtia/main/compose.own-ollama.yaml
docker compose -f compose.own-ollama.yaml up

MYRTIA_OLLAMA_URL=http://gpu-server:11434 docker compose -f compose.own-ollama.yaml up

The first run takes a few minutes: the council plugin pulls in IBM ARES core (ares-redteamer), which brings PyTorch and Transformers. After that, cd myrtia && uv run myrtia starts in seconds. Options: uv run myrtia --port 8010 --no-browser.

What you get

uv run myrtia gives you 21 attacks, ready to launch. 17 are built into Myrtia; the council pair comes from the bundled ARES Council pack, installed by default, plus NVIDIA Garak and Microsoft PyRIT pack adapters.

The garak-style probes are Myrtia's own implementations — they don't need the garak library. The full NVIDIA garak and Microsoft PyRIT libraries are optional packs under Plugins in the console. Installing a pack adds a real attack driven by that library, graded 0-5 by the same evaluator as everything else, so it feeds the ASR dashboard and audit reports:

• garak — run any garak probe's real prompt corpus (encoding.InjectBase64, dan.DanInTheWildFull, latentinjection.LatentInjectionResume, …) against the target.
• pyrit — transform each behaviour with a real PyRIT prompt converter (Base64, ROT13, leetspeak, Morse, Atbash, Caesar, binary, NATO, emoji) and ask the target to decode and comply.

See Attacks for what each one does.

Where your models run

Myrtia itself is light: it only makes HTTP calls to OpenAI-compatible endpoints. The models (council, chairman, judge, target) run in Ollama, vLLM or a hosted API, and each role can point at its own endpoint — a local Ollama, or a GPU server elsewhere on your network.

Using the console

Pick an attack, choose a target (OpenAI-compatible base URL and model), select behaviours from the library or type a goal, and launch. The deliberation streams in live and the run lands in History. Only runs marked Security evaluation feed the Statistics dashboard and audit exports by default; the harmless starter goal is a Smoke test. Older runs remain Unreviewed until you classify them in the run detail. Judge scores are automatic signals, not confirmed vulnerabilities.

HarmBench goals are included. In Plugins → Goal datasets, download JailbreakBench, AdvBench or StrongREJECT to add their goals to both run modes. Search and filter by dataset and category in Runs. See goal dataset sources and scope.

cd myrtia/backend
python3 -m venv .venv && source .venv/bin/activate
pip install -e . -e vendor/ares-council
uvicorn myrtia.api.app:app --port 8000

Something not working? See Troubleshooting.

Table of Contents

• Quick Start
• Overview
• Why Myrtia
• Features
• Attacker topologies
• Current Capability Boundaries
• Architecture
• Troubleshooting
• Attacks
• Configuration
• Extending Myrtia
• Red teaming as a service
• Roadmap
• Built On
• Responsible Use
• License

Overview

Myrtia is an open-source framework for red teaming large language models and AI agents. Rather than a single attacker, it orchestrates a council of models that independently propose adversarial prompts, cross-critique one another, and vote (via Borda aggregation) on the strongest candidate. A chairman synthesises the winning proposal, the resulting prompt is sent to the target model, and an in-loop judge grades the outcome on a 0–5 harm scale. The entire deliberation is streamed to a web interface in real time, turn by turn, and every run rolls up into Attack Success Rate (ASR) analytics.

Library and CLI tools such as IBM ARES and NVIDIA garak are the reference point for LLM red teaming. Myrtia takes the same rigour but is built around autonomous multi-agent attackers, and it ships with a real front-end and analytics so that operators can observe an attack as it unfolds and reason about why a target held or failed, not just whether it did.

How Myrtia compares

The two bold rows are Myrtia's differentiators: no other tool pairs a deliberating council with Borda voting and chairman synthesis with an open-source visual canvas for composing the agentic attack. The closest neighbours sit in only one dimension — multi-agent attacker research (e.g. X-Teaming, a fixed Planner/Attacker/Verifier pipeline) or generic visual agent builders (LangGraph Studio, Langflow), which are not red-teaming tools. Agentic Radar is complementary rather than a competitor: it statically scans agent source code (LangGraph, CrewAI, n8n, OpenAI Agents, AutoGen) to map workflows, tools and MCP servers to OWASP risks, which Myrtia does not do; its runtime test sends a fixed set of probes to OpenAI Agents only. (SaaS platforms compared from their public descriptions.)

The name and the mark

The name Myrtia comes from Myrtea / Murtea, "place of myrtles" — one of the leading hypotheses for the origin of the name of the city of Murcia (Fernández Nieto & Molina Gómez, 2006); the same myrtle that Varro, in Rome, linked to the goddess later known as Venus Murcia. The city was founded as Madinat Mursiya in 825, and its name may carry that older heritage. It is a tool from the huerta, aimed at the frontier.

The logo carries the same idea in one image. It is a myrtle leaf — the plant that gives Myrtia and Murcia their name, our roots in the huerta — and, at the same time, a shield cracked open by our attacks. Defence and offence in a single mark: the shield is what an aligned model puts up, and the break is what a deliberating council of attackers finds in it. The heritage is deliberately Spanish — a rodela, the round shield of the Tercios — reclaimed for AI security research from Murcia, in the spirit of turning a frontier tool outward at the frontier models. A tool from the huerta, forged as a broken shield, pointed at the strongest systems we can reach.

Red teaming in Myrtia is organised around four components:

• Attacks define how adversarial prompts are generated: a deliberating council, a crescendo escalation, or a direct baseline.
• Targets are the model or agent under test, reached over HTTP: OpenAI-compatible endpoints (Ollama, vLLM, OpenAI, any proxy), Anthropic's native Messages API, or any JSON agent endpoint.
• Judge and scoring grade each response on a graded 0–5 scale and aggregate it into ASR.
• Behaviours are what you attack: a built-in HarmBench library selectable by theme, or your own goals.

Why Myrtia

• Deliberation beats a lone attacker. Multiple models proposing and critiquing surface attack angles a single model misses; Borda voting selects the strongest, and a chairman refines it before it ever reaches the target.
• Observability is the product. Red-team results are only trustworthy if they are legible and reproducible. Myrtia shows every proposal, every vote, every judge rationale.
• Attacker/target separation. Safety-aligned frontier models refuse to author jailbreaks, so Myrtia pairs a permissive open-model council with the frontier target under test — the realistic threat model, and the one that actually produces signal.
• Permissive and self-hosted. Apache-2.0, runs on your own machine, mixes any providers, and never returns API keys through the API or the event stream.

Features

• Live CouncilAttack. N proposer models propose, cross-critique, vote (Borda Top-1), a chairman synthesises, and an in-loop judge grades — rendered live over WebSocket, turn by turn.
• Multiple attack strategies. council, council-crescendo (accumulating multi-turn escalation), and direct (baseline), each exposing its own advanced configuration in the UI.
• Behaviour library. 278 HarmBench behaviours across five themes, selectable via per-theme checkboxes and search, or replaced entirely by your own goals.
• Provider-agnostic connectors. OpenAI, Anthropic (OpenAI-compatible endpoint), self-hosted vLLM, and local Ollama, mixed freely across council, chairman, judge, and target roles.
• Analytics dashboard. ASR broken down by attack, target model, and theme; judge-score distribution; and, for council attacks, per-member vote win-rates and average Borda scores.
• Searchable run history. Every run is filterable by goal text, theme, result, and evaluation score, and can be reopened to replay its full deliberation.
• Legible deliberation view. Per-goal and per-turn cards, Borda vote tables, and read-more and collapse controls throughout, in dark or light themes.
• Plugin architecture. Attacks, targets, and judges are plugins; the attack contract is ARES-compatible, so existing ARES strategies can be hosted with minimal adaptation.
• Security-conscious by default. API keys are supplied at run time and are redacted from all API responses and from the live event stream.
• Composable attacker topologies. Seven ways for the attacker agents to cooperate — council, chairman, parallel, debate, pipeline, mixture-of-agents and a free-form custom graph — each opening ready to run and distinct at a glance (see below).
• Self-hosted and permissive. Apache-2.0, single-command backend, no vendor lock-in.

Attacker topologies

The visual builder is not a fixed council — it is a canvas for how your attacker agents cooperate. Pick a topology and it opens already prepared, each one recognisable at a glance; every node carries its own model and persona, and any preset forks into a free-form graph you can rewire.

Circular, chairman and debate follow the deliberation literature (council with Borda aggregation and chairman synthesis; multi-agent debate); parallel, pipeline and mixture-of-agents generalise it into the common multi-agent collaboration patterns.

Current Capability Boundaries

Myrtia is in alpha. Being explicit about what it is and is not today:

• Attack coverage is CouncilAttack (isolated and crescendo) plus a direct baseline. The broader ARES/garak catalogue (echo-chamber, TAP, AutoDAN, GCG, and others) is planned, not shipped.
• Storage is in-memory and single-process. Run history and the statistics dashboard reset on backend restart; a durable SQLite backing is on the roadmap.
• There is no run cancellation and no built-in authentication yet; Myrtia is intended for local, single-operator use behind your own controls.
• Judge quality is load-bearing: a weak judge inflates ASR with false positives. Use a capable model as judge, and treat scores from small local judges as indicative only.
• Myrtia is a red-teaming tool for probing models and agents. It is not a network penetration testing platform and ships no offensive network tooling.

Architecture

flowchart TB
    op["Security engineer"]
    myrtia["Myrtia<br/>agent-native red-teaming"]
    target["Target<br/>LLM or agent under test"]
    council["Council, chairman, judge<br/>OpenAI / Anthropic / vLLM / Ollama"]

    op -->|launches runs, watches live| myrtia
    myrtia -->|attack prompts| target
    myrtia -->|proposals, critiques, synthesis, judging| council
    target -->|responses| myrtia

flowchart LR
    G[Behaviour / goal] --> P
    subgraph Turn
      P[Council members propose] --> C[Cross-critique]
      C --> V[Borda Top-1 vote]
      V --> CH[Chairman synthesise]
      CH --> T[Send to target]
      T --> J[Graded 0-5 judge]
    end
    J -->|score below 5 and turns remaining| P
    J -->|score = 5 or budget spent| R[Result and ASR]

flowchart LR
    ORCH[Orchestrator] -->|typed events| BUS[(In-memory event bus<br/>per-run backlog)]
    BUS -->|WebSocket replay| UI[Web UI live deliberation]
    API[FastAPI REST] -->|create run, history, stats| UI

backend/   Python, FastAPI + WebSocket
  myrtia/orchestrator   drives a run, streams events
  myrtia/plugins        the 19 attacks (ARES-compatible): council, graph, crescendo, tap, probes…
  myrtia/connectors     OpenAI-compatible target connectors
  myrtia/events         typed event schema + thread-safe bus (the live-view contract)
  myrtia/judges         graded harm judge and scoring
  myrtia/data           HarmBench behaviour library
  myrtia/api            REST control + /runs/{id}/events WebSocket
  vendor/ares-council   vendored CouncilAttack plugin (Apache-2.0)
frontend/  web UI (live deliberation, history, behaviour library, dashboard)
docs/      architecture and reference maps

See docs/architecture.md for the full design.

Troubleshooting

In the console, failed turns now show a short explanation (e.g. "Can't reach the model endpoint") with the raw error tucked under Details.

Configuration

Recommended setup — the council configuration from the paper: a local open-weight council attacking a frontier target.

These are the models the paper's attacks were run with. Smaller models work for a quick smoke test, but the open-weight council above is what reproduces the reported results.

Any role can point at any OpenAI-compatible endpoint. Council, chairman, and judge models, turn budget, Borda voting, and chairman synthesis are all configurable per run (the UI reads each attack's configuration schema and renders the advanced options automatically).

Attacks

Myrtia ships two kinds of attack. Some drive an attacker-side LLM (a council, a single helper, or an iterative attacker) and so take a model per role; the rest are static transforms or template libraries that need no attacker model — only the shared judge scores the target's reply.

Recommended models per role

The defaults are demo-sized so a laptop with a 2–3 B model works out of the box. When you have the hardware (a 24–80 GB GPU, multi-GPU, or Apple-silicon 64 GB+), use the Recommended column. Our paper is the exact roster from the CouncilAttack / ContextualJailbreak experiments — reproduce those numbers by setting these.

About deepseek-r1:32b — it is DeepSeek-R1-Distill-Qwen-32B: Qwen2.5-32B fine-tuned (SFT, no RL) on reasoning traces distilled from the 671 B DeepSeek-R1 teacher. Default Ollama quant is Q4_K_M (~19.85 GB). Pull with ollama pull deepseek-r1:32b (model card). A heterogeneous council (different families: Qwen, Gemma, a reasoning distill) is deliberate — diverse members surface complementary attack angles.

Set these per role in the attack config, e.g. for crescendo:

{
  "helper_model": "deepseek-r1:32b",
  "judge_model": "gpt-oss:120b",
  "aux_endpoint": "http://127.0.0.1:11434/v1"
}

Extending Myrtia

Attacks are ARES-style strategies: a class constructed with (config, target_connector) and exposing an attack() method, registered as a Myrtia plugin. A plugin may add an optional on_event seam for live streaming and a config_schema so the UI can render its advanced options. Targets are ARES connectors built from a small spec. To add an attack, drop a module in backend/myrtia/plugins/ and register it in the plugin registry.

Red teaming as a service

Myrtia is open source today. The direction is a hosted service: point your own model and agent endpoints at a deliberating council and get a continuous, reproducible adversarial audit — ASR trends, regressions and an exportable report — without standing up the infrastructure yourself. The open-source console is the same engine that service is built on. (On the roadmap.)

Roadmap

Development is tracked in BACKLOG.md, organised into sprints with known issues and proposals.

Built On

Myrtia builds on the ARES red-teaming abstractions (connectors and attack strategies) and vendors the CouncilAttack plugin. It is inspired by the LLM red-teaming work of ARES (IBM) and garak (NVIDIA).

Responsible Use

Myrtia is intended for authorised AI safety and security testing: evaluating models and agents you own or have explicit permission to test. Do not use it to produce or distribute harmful content, or to attack systems without authorisation.

License

Apache-2.0.