vllm-project/agentic-api
vllm-project/agentic-api has 355 GitHub stars and 106 forks. GitHot measured +2 stars in 24 hours. Last checked 2026-11-17 03:53 UTC. Last push 2026-11-16.
What you can do
- ๐ Stateful conversations: the server manages history through previous_response_id or the Conversations API, so clients stop replaying full transcripts.
- ๐ ๏ธ Server-side tools: an explicit ownership model decides what runs where. Web search (You.com, Brave, Tavily, Serply, Keenable, or SearXNG), web fetch, MCP, file search, and an opt-in embedded code interpreter run on the gateway.
- ๐ค Multi-agent orchestration: one Responses request can spawn and coordinate subagents server-side (details).
- ๐ก Every transport: non-streaming HTTP, SSE streaming, and WebSocket with stream_id multiplexing.
Measured star growth
- in 24h
- +2
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Stateful API logic for agentic applications using vLLM
AIREADME.md
OriginalThe stateful, agentic API layer for vLLM, written in Rust ๐ฆ
Run OpenAI-grade agentic workloads (Responses and Messages APIs, server-side tools, Codex, Claude Code) on your own GPUs.
๐ง Overview
vLLM gives you state-of-the-art inference throughput. But real agentic applications need more than raw tokens: they need conversation state, tool-call loops, and multi-turn orchestration. Today, all of that complexity lives in your client code.
Agentic API moves it server-side. It is a Rust-native gateway that sits in front of vLLM and owns the stateful agentic APIs, starting with an OpenAI-compatible Responses API. vLLM is one supported backend, not part of the Agentic API product name. Your application makes one API call and the server handles the rest: state hydration, tool execution, streaming, and continuation.
flowchart LR
C(["๐งโ๐ป Client<br/>Codex ยท Claude Code ยท SDKs ยท curl"]) -->|"๐ฎ <code>POST /v1/responses</code> ยท <code>/v1/messages</code><br/>๐ HTTP ๐ก SSE ๐ WebSocket"| A
subgraph A ["โก Agentic API (Rust ๐ฆ)"]
direction TB
S["๐ State hydration<br/><code>previous_response_id</code>"]
T["๐ ๏ธ Server-side tools<br/>web search ยท web fetch ยท MCP ยท code interpreter"]
P["๐พ Persistence<br/>SQLite ยท PostgreSQL"]
end
A -->|"๐ <code>POST /v1/responses</code> ยท <code>/v1/messages</code><br/>โ๏ธ stateless ๐ค OpenAI/Anthropic-compatible"| V(["๐ vLLM core<br/>inference engine"])
classDef client fill:#FFE8B3,stroke:#F59E0B,stroke-width:2px,color:#7C2D12
classDef inner fill:#E0E7FF,stroke:#6366F1,stroke-width:2px,color:#312E81
classDef engine fill:#DCFCE7,stroke:#22C55E,stroke-width:2px,color:#14532D
class C client
class S,T,P inner
class V engine
style A fill:#F5F3FF,stroke:#8B5CF6,stroke-width:2px,color:#5B21B6
linkStyle 0 stroke:#F59E0B,stroke-width:2px
linkStyle 1 stroke:#22C55E,stroke-width:2px
[!TIP] Point OpenAI Codex at Agentic API and drive it entirely with open models served by vLLM. No OpenAI account required.
โจ Key Features
- ๐ Stateful conversations: the server manages history through
previous_response_idor the Conversations API, so clients stop replaying full transcripts. - ๐ ๏ธ Server-side tools: an explicit ownership model decides what runs where. Web search (You.com, Brave, Tavily, Serply, Keenable, or SearXNG), web fetch, MCP, file search, and an opt-in embedded code interpreter run on the gateway.
- ๐ค Multi-agent orchestration: one Responses request can spawn and coordinate subagents server-side (details).
- ๐ก Every transport: non-streaming HTTP, SSE streaming, and WebSocket with
stream_idmultiplexing. - ๐งฐ Codex and Claude Code ready: Codex-shaped Responses traffic and Anthropic Messages traffic keep the tool declarations and item shapes each client depends on.
- ๐๏ธ Compaction: automatic and explicit context compaction through
/v1/responses/compact(guide). - ๐ญ Observability: opt-in OpenTelemetry traces and metrics over OTLP (guide).
- โ Compatibility tested: validated against the Open Responses suite, with replay-cassette tests for OpenAI, vLLM, SGLang, and NVIDIA Dynamo traffic.
๐งญ API Surface
| Endpoint | Description |
|---|---|
POST /v1/responses ยท GET /v1/responses (WebSocket) |
Responses API with state, tools, and streaming |
GET ยท DELETE /v1/responses/{id} ยท GET /v1/responses/{id}/input_items |
Retrieve, delete, and list the input of a stored response |
POST /v1/responses/compact |
Compact direct input or a stored response chain |
/v1/conversations ยท /v1/conversations/{id}/items |
Conversations API |
POST /v1/messages ยท POST /v1/messages/count_tokens |
Anthropic Messages API with a server-side loop for gateway-owned tools |
/v1/files ยท /v1/vector_stores |
Files and vector stores for the file_search tool (guide) |
GET /v1/models ยท GET /v1/models/{model} |
Model listing from the upstream |
/v1/chat/completions ยท /v1/completions ยท /v1/embeddings ยท /v1/audio/* |
Forwarded to the upstream unchanged |
GET /health ยท GET /ready |
Liveness and readiness probes |
GET /swagger-ui ยท GET /openapi.json |
OpenAPI spec, served when ENABLE_OPENAPI_DOCS=true |
Planned: stateful Messages and the Interactions API. See the API reference for routing, authentication, retrieval, and WebSocket details.
๐ Quickstart
1. Serve a model with vLLM. Any recipe from recipes.vllm.ai works:
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
--tool-call-parser qwen3_coder --enable-auto-tool-choice \
--reasoning-parser qwen3 --port 5050
NVIDIA Dynamo and SGLang upstreams work too.
2. Start Agentic API in front of it. Web search is optional; pick a provider and set its key:
AGENTIC_WEB_SEARCH_PROVIDER=brave BRAVE_API_KEY=<your-brave-api-key> \
cargo run -p agentic-server -- --llm-api-base http://127.0.0.1:5050
Or use the published package instead of building (see Python installation).
3. Make a stateful call:
curl http://localhost:9000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-30B-A3B-FP8",
"input": "What is new in vLLM this month?",
"tools": [{"type": "web_search"}]
}'
Pass the returned id as previous_response_id to continue, and the server rehydrates the history for you.
One command for Codex or Claude Code
The agentic CLI starts the gateway and launches a harness with an isolated configuration:
cargo build -p agentic-server --bins
./target/debug/agentic run codex --upstream http://127.0.0.1:5050
./target/debug/agentic run claude --upstream http://127.0.0.1:5050 --model Qwen/Qwen3-30B-A3B-FP8
./target/debug/agentic validate --upstream http://127.0.0.1:5050 --harness codex # preflight checks only
The model defaults to the first one the upstream lists. SQLite is the default store; pass --database-url postgresql://... for a shared session. See the quickstart guide and
harness CLI testing, including
Claude Code with a self-hosted model.
๐ Python installation
The agentic-api package on PyPI bundles the Rust gateway, the
agentic CLI, and a small Python launcher. Wheels cover Linux x86_64 (glibc 2.17+), macOS Intel, and macOS Apple
Silicon on Python 3.10+.
uvx --from agentic-api==0.10.0 agentic --version
uvx --from agentic-api==0.10.0 agentic serve --upstream http://existing-vllm:8000
python -m pip install agentic-api==0.10.0 # remote mode, no vLLM
python -m pip install "agentic-api[local]==0.10.0" # also installs the pinned vLLM runtime
With uv:
uv pip install agentic-api==0.10.0
uv pip install "agentic-api[local]==0.10.0"
uvx --from agentic-api==0.10.0 agentic-api doctor
uvx --from agentic-api==0.10.0 agentic-api serve --vllm-base-url http://existing-vllm:8000
See Python installation and workflows for managed vLLM, doctor output, and
known-good model profiles.
๐ค Codex on your own GPUs
Agentic API speaks the Responses wire protocol Codex expects, including WebSockets. Add a provider to
~/.codex/config.toml:
[model_providers.agentic-api]
name = "OpenAI"
base_url = "http://localhost:9000/v1"
wire_api = "responses"
requires_openai_auth = false
supports_websockets = true
codex --disable image_generation -c model_provider=agentic-api -m Qwen/Qwen3-30B-A3B-FP8
For the desktop app, see Codex Desktop with local models. With OIDC enabled, replace
requires_openai_auth = false with Codex's command-backed bearer authentication; see
GitHub authentication with Dex.
๐งโ๐ป Claude Code on your own GPUs
Agentic API serves the Anthropic Messages protocol at /v1/messages, so Claude Code (CLI or Agent SDK) runs against
open models:
export ANTHROPIC_BASE_URL="http://localhost:9000"
export ANTHROPIC_API_KEY="<your-key>"
export ANTHROPIC_MODEL="Qwen/Qwen3-30B-A3B-FP8" # match the served model
claude -p "summarize the files in this directory"
Claude Code's own tools stay client-owned. Its native web search and web fetch tools run on the gateway; see
Gateway server tools on /v1/messages. With OIDC enabled, send the token in
ANTHROPIC_AUTH_TOKEN and leave ANTHROPIC_API_KEY unset (GitHub authentication with Dex).
โ๏ธ Configuration
Agentic API reads ~/.agentic-api/config.toml and generates it on first run. CLI arguments and environment variables
take precedence over the file, so secrets can stay in the environment:
llm_api_base = "http://127.0.0.1:5050"
[web_search]
provider = "brave"
- Configuration: storage, request and rate limits, model capabilities, response limits, and MCP servers.
- Responses capability policies: per-model parameter limits, prompt-cache controls, and service tiers.
- Web search providers: provider settings and how each one maps the shared tool contract.
- Embedded code interpreter: the opt-in
embedded-code-interpreterbuild feature. - Deploying: containers, Kubernetes, and observability.
๐งฉ Tool Ownership
Every tool call has exactly one execution path, so nothing runs by accident:
| Ownership | Who executes it | Examples |
|---|---|---|
| Gateway-owned | Agentic API, which then continues the loop | Web search, web fetch, MCP, file search, code interpreter (opt-in) |
| Client-owned | Returned to the client | Shell tool, Codex editor tools, your functions |
| Provider-owned | Passed through to vLLM or an upstream provider | Provider-native tools |
Unknown or ambiguous tool shapes are never executed by default. They are preserved and returned.
๐ค Multi-agent Responses
Set multi_agent.enabled and the gateway runs the
Responses multi-agent loop itself: the root
agent spawns subagents, each with its own history and tools, and the response returns their attributed
multi_agent_call, multi_agent_call_output, and agent_message items.
curl http://localhost:9000/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-30B-A3B-FP8",
"store": true,
"multi_agent": {"enabled": true, "max_concurrent_subagents": 3},
"input": "Have one agent review correctness and another review test coverage, then summarize."
}'
It works over HTTP, SSE, and WebSocket. Continue a stored agent tree with previous_response_id; with store: false,
send the complete item history on each request. See the API reference and
multi-agent execution for the details and limits.
๐๏ธ Repository Layout
crates/
โโโ agentic-server/ # Axum binary, transport handlers (HTTP/SSE/WS), configuration, `agentic` CLI
โโโ agentic-server-core/ # Protocol types, executor, tool framework, persistence
โโโ agentic-llm-d/ # Split-execution state backend for the llm-d coordinator
โโโ agentic-cli-docs/ # Build-time generator for the CLI reference docs
โโโ agentic-praxis/ # Praxis gateway integration
python/agentic_api/ # Python distribution and launcher for the packaged gateway
docs/ # MkDocs documentation, ADRs, and design notes
๐ ๏ธ Developing
cargo build # build
cargo test # test
cargo clippy --all-targets -- -D warnings # lint
cargo fmt -- --check # format check
uv run mkdocs serve # docs (after: uv pip install -r docs/requirements.txt)
Read ARCHITECTURE and CONTRIBUTING before opening a PR. Design decisions live in docs/adr/ and docs/design/.
๐บ๏ธ Roadmap at a Glance
- Responses API hydration: stateful continuation with
previous_response_id - Codex support: practical Codex sessions through the Responses API
- Server-side tool execution: explicit ownership, web search, web fetch, MCP, and code interpreter built in
- Messages API: Claude Code support with a server-side gateway tool loop
- Conversations API: OpenAI-compatible conversation and item management
- Multi-agent Responses: stateful and stateless execution over HTTP and WebSocket
- File search: Files and vector-store APIs with a gateway-executed
file_searchtool - Stateful Messages: Messages continuation on the same persistence and execution primitives
- Interactions API: durable, higher-level agentic workflows
- Production hardening: tenant-scoped state and cached-prefix continuation
๐ License
Licensed under the Apache License 2.0.
โญ If Agentic API saves you from writing one more client-side tool loop, star the repo! โญ