Skip to content
Projects

vllm-project/agentic-api has 355 GitHub stars and 106 forks. GitHot measured +2 stars in 24 hours. Last checked 2026-11-17 03:53 UTC. Last push 2026-11-16.

How these counts are measured

What you can do

  • ๐Ÿ”„ Stateful conversations: the server manages history through previous_response_id or the Conversations API, so clients stop replaying full transcripts.
  • ๐Ÿ› ๏ธ Server-side tools: an explicit ownership model decides what runs where. Web search (You.com, Brave, Tavily, Serply, Keenable, or SearXNG), web fetch, MCP, file search, and an opt-in embedded code interpreter run on the gateway.
  • ๐Ÿค Multi-agent orchestration: one Responses request can spawn and coordinate subagents server-side (details).
  • ๐Ÿ“ก Every transport: non-streaming HTTP, SSE streaming, and WebSocket with stream_id multiplexing.

Measured star growth

in 24h
+2
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Stateful API logic for agentic applications using vLLM

AI

README.md

Original
Agentic API

The stateful, agentic API layer for vLLM, written in Rust ๐Ÿฆ€

Run OpenAI-grade agentic workloads (Responses and Messages APIs, server-side tools, Codex, Claude Code) on your own GPUs.

License Rust CI pre-commit


๐Ÿง  Overview

vLLM gives you state-of-the-art inference throughput. But real agentic applications need more than raw tokens: they need conversation state, tool-call loops, and multi-turn orchestration. Today, all of that complexity lives in your client code.

Agentic API moves it server-side. It is a Rust-native gateway that sits in front of vLLM and owns the stateful agentic APIs, starting with an OpenAI-compatible Responses API. vLLM is one supported backend, not part of the Agentic API product name. Your application makes one API call and the server handles the rest: state hydration, tool execution, streaming, and continuation.

flowchart LR
    C(["๐Ÿง‘โ€๐Ÿ’ป Client<br/>Codex ยท Claude Code ยท SDKs ยท curl"]) -->|"๐Ÿ“ฎ <code>POST /v1/responses</code> ยท <code>/v1/messages</code><br/>๐ŸŒ HTTP&nbsp;&nbsp;๐Ÿ“ก SSE&nbsp;&nbsp;๐Ÿ”Œ WebSocket"| A
    subgraph A ["โšก Agentic API (Rust ๐Ÿฆ€)"]
        direction TB
        S["๐Ÿ”„ State hydration<br/><code>previous_response_id</code>"]
        T["๐Ÿ› ๏ธ Server-side tools<br/>web search ยท web fetch ยท MCP ยท code interpreter"]
        P["๐Ÿ’พ Persistence<br/>SQLite ยท PostgreSQL"]
    end
    A -->|"๐Ÿš€ <code>POST /v1/responses</code> ยท <code>/v1/messages</code><br/>โš™๏ธ stateless&nbsp;&nbsp;๐Ÿค OpenAI/Anthropic-compatible"| V(["๐Ÿš€ vLLM core<br/>inference engine"])

    classDef client fill:#FFE8B3,stroke:#F59E0B,stroke-width:2px,color:#7C2D12
    classDef inner fill:#E0E7FF,stroke:#6366F1,stroke-width:2px,color:#312E81
    classDef engine fill:#DCFCE7,stroke:#22C55E,stroke-width:2px,color:#14532D

    class C client
    class S,T,P inner
    class V engine
    style A fill:#F5F3FF,stroke:#8B5CF6,stroke-width:2px,color:#5B21B6
    linkStyle 0 stroke:#F59E0B,stroke-width:2px
    linkStyle 1 stroke:#22C55E,stroke-width:2px

[!TIP] Point OpenAI Codex at Agentic API and drive it entirely with open models served by vLLM. No OpenAI account required.

โœจ Key Features

  • ๐Ÿ”„ Stateful conversations: the server manages history through previous_response_id or the Conversations API, so clients stop replaying full transcripts.
  • ๐Ÿ› ๏ธ Server-side tools: an explicit ownership model decides what runs where. Web search (You.com, Brave, Tavily, Serply, Keenable, or SearXNG), web fetch, MCP, file search, and an opt-in embedded code interpreter run on the gateway.
  • ๐Ÿค Multi-agent orchestration: one Responses request can spawn and coordinate subagents server-side (details).
  • ๐Ÿ“ก Every transport: non-streaming HTTP, SSE streaming, and WebSocket with stream_id multiplexing.
  • ๐Ÿงฐ Codex and Claude Code ready: Codex-shaped Responses traffic and Anthropic Messages traffic keep the tool declarations and item shapes each client depends on.
  • ๐Ÿ—œ๏ธ Compaction: automatic and explicit context compaction through /v1/responses/compact (guide).
  • ๐Ÿ”ญ Observability: opt-in OpenTelemetry traces and metrics over OTLP (guide).
  • โœ… Compatibility tested: validated against the Open Responses suite, with replay-cassette tests for OpenAI, vLLM, SGLang, and NVIDIA Dynamo traffic.

๐Ÿงญ API Surface

Endpoint Description
POST /v1/responses ยท GET /v1/responses (WebSocket) Responses API with state, tools, and streaming
GET ยท DELETE /v1/responses/{id} ยท GET /v1/responses/{id}/input_items Retrieve, delete, and list the input of a stored response
POST /v1/responses/compact Compact direct input or a stored response chain
/v1/conversations ยท /v1/conversations/{id}/items Conversations API
POST /v1/messages ยท POST /v1/messages/count_tokens Anthropic Messages API with a server-side loop for gateway-owned tools
/v1/files ยท /v1/vector_stores Files and vector stores for the file_search tool (guide)
GET /v1/models ยท GET /v1/models/{model} Model listing from the upstream
/v1/chat/completions ยท /v1/completions ยท /v1/embeddings ยท /v1/audio/* Forwarded to the upstream unchanged
GET /health ยท GET /ready Liveness and readiness probes
GET /swagger-ui ยท GET /openapi.json OpenAPI spec, served when ENABLE_OPENAPI_DOCS=true

Planned: stateful Messages and the Interactions API. See the API reference for routing, authentication, retrieval, and WebSocket details.

๐Ÿš€ Quickstart

1. Serve a model with vLLM. Any recipe from recipes.vllm.ai works:

vllm serve Qwen/Qwen3-30B-A3B-FP8 \
  --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --reasoning-parser qwen3 --port 5050

NVIDIA Dynamo and SGLang upstreams work too.

2. Start Agentic API in front of it. Web search is optional; pick a provider and set its key:

AGENTIC_WEB_SEARCH_PROVIDER=brave BRAVE_API_KEY=<your-brave-api-key> \
  cargo run -p agentic-server -- --llm-api-base http://127.0.0.1:5050

Or use the published package instead of building (see Python installation).

3. Make a stateful call:

curl http://localhost:9000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "input": "What is new in vLLM this month?",
    "tools": [{"type": "web_search"}]
  }'

Pass the returned id as previous_response_id to continue, and the server rehydrates the history for you.

One command for Codex or Claude Code

The agentic CLI starts the gateway and launches a harness with an isolated configuration:

cargo build -p agentic-server --bins
./target/debug/agentic run codex --upstream http://127.0.0.1:5050
./target/debug/agentic run claude --upstream http://127.0.0.1:5050 --model Qwen/Qwen3-30B-A3B-FP8
./target/debug/agentic validate --upstream http://127.0.0.1:5050 --harness codex   # preflight checks only

The model defaults to the first one the upstream lists. SQLite is the default store; pass --database-url postgresql://... for a shared session. See the quickstart guide and harness CLI testing, including Claude Code with a self-hosted model.

๐Ÿ Python installation

The agentic-api package on PyPI bundles the Rust gateway, the agentic CLI, and a small Python launcher. Wheels cover Linux x86_64 (glibc 2.17+), macOS Intel, and macOS Apple Silicon on Python 3.10+.

uvx --from agentic-api==0.10.0 agentic --version
uvx --from agentic-api==0.10.0 agentic serve --upstream http://existing-vllm:8000

python -m pip install agentic-api==0.10.0                # remote mode, no vLLM
python -m pip install "agentic-api[local]==0.10.0"       # also installs the pinned vLLM runtime

With uv:

uv pip install agentic-api==0.10.0
uv pip install "agentic-api[local]==0.10.0"
uvx --from agentic-api==0.10.0 agentic-api doctor
uvx --from agentic-api==0.10.0 agentic-api serve --vllm-base-url http://existing-vllm:8000

See Python installation and workflows for managed vLLM, doctor output, and known-good model profiles.

๐Ÿค– Codex on your own GPUs

Agentic API speaks the Responses wire protocol Codex expects, including WebSockets. Add a provider to ~/.codex/config.toml:

[model_providers.agentic-api]
name = "OpenAI"
base_url = "http://localhost:9000/v1"
wire_api = "responses"
requires_openai_auth = false
supports_websockets = true
codex --disable image_generation -c model_provider=agentic-api -m Qwen/Qwen3-30B-A3B-FP8

For the desktop app, see Codex Desktop with local models. With OIDC enabled, replace requires_openai_auth = false with Codex's command-backed bearer authentication; see GitHub authentication with Dex.

๐Ÿง‘โ€๐Ÿ’ป Claude Code on your own GPUs

Agentic API serves the Anthropic Messages protocol at /v1/messages, so Claude Code (CLI or Agent SDK) runs against open models:

export ANTHROPIC_BASE_URL="http://localhost:9000"
export ANTHROPIC_API_KEY="<your-key>"
export ANTHROPIC_MODEL="Qwen/Qwen3-30B-A3B-FP8"   # match the served model

claude -p "summarize the files in this directory"

Claude Code's own tools stay client-owned. Its native web search and web fetch tools run on the gateway; see Gateway server tools on /v1/messages. With OIDC enabled, send the token in ANTHROPIC_AUTH_TOKEN and leave ANTHROPIC_API_KEY unset (GitHub authentication with Dex).

โš™๏ธ Configuration

Agentic API reads ~/.agentic-api/config.toml and generates it on first run. CLI arguments and environment variables take precedence over the file, so secrets can stay in the environment:

llm_api_base = "http://127.0.0.1:5050"

[web_search]
provider = "brave"

๐Ÿงฉ Tool Ownership

Every tool call has exactly one execution path, so nothing runs by accident:

Ownership Who executes it Examples
Gateway-owned Agentic API, which then continues the loop Web search, web fetch, MCP, file search, code interpreter (opt-in)
Client-owned Returned to the client Shell tool, Codex editor tools, your functions
Provider-owned Passed through to vLLM or an upstream provider Provider-native tools

Unknown or ambiguous tool shapes are never executed by default. They are preserved and returned.

๐Ÿค Multi-agent Responses

Set multi_agent.enabled and the gateway runs the Responses multi-agent loop itself: the root agent spawns subagents, each with its own history and tools, and the response returns their attributed multi_agent_call, multi_agent_call_output, and agent_message items.

curl http://localhost:9000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B-FP8",
    "store": true,
    "multi_agent": {"enabled": true, "max_concurrent_subagents": 3},
    "input": "Have one agent review correctness and another review test coverage, then summarize."
  }'

It works over HTTP, SSE, and WebSocket. Continue a stored agent tree with previous_response_id; with store: false, send the complete item history on each request. See the API reference and multi-agent execution for the details and limits.

๐Ÿ—๏ธ Repository Layout

crates/
โ”œโ”€โ”€ agentic-server/       # Axum binary, transport handlers (HTTP/SSE/WS), configuration, `agentic` CLI
โ”œโ”€โ”€ agentic-server-core/  # Protocol types, executor, tool framework, persistence
โ”œโ”€โ”€ agentic-llm-d/        # Split-execution state backend for the llm-d coordinator
โ”œโ”€โ”€ agentic-cli-docs/     # Build-time generator for the CLI reference docs
โ””โ”€โ”€ agentic-praxis/       # Praxis gateway integration
python/agentic_api/       # Python distribution and launcher for the packaged gateway
docs/                     # MkDocs documentation, ADRs, and design notes

๐Ÿ› ๏ธ Developing

cargo build                                  # build
cargo test                                   # test
cargo clippy --all-targets -- -D warnings    # lint
cargo fmt -- --check                         # format check
uv run mkdocs serve                          # docs (after: uv pip install -r docs/requirements.txt)

Read ARCHITECTURE and CONTRIBUTING before opening a PR. Design decisions live in docs/adr/ and docs/design/.

๐Ÿ—บ๏ธ Roadmap at a Glance

  • Responses API hydration: stateful continuation with previous_response_id
  • Codex support: practical Codex sessions through the Responses API
  • Server-side tool execution: explicit ownership, web search, web fetch, MCP, and code interpreter built in
  • Messages API: Claude Code support with a server-side gateway tool loop
  • Conversations API: OpenAI-compatible conversation and item management
  • Multi-agent Responses: stateful and stateless execution over HTTP and WebSocket
  • File search: Files and vector-store APIs with a gateway-executed file_search tool
  • Stateful Messages: Messages continuation on the same persistence and execution primitives
  • Interactions API: durable, higher-level agentic workflows
  • Production hardening: tenant-scoped state and cached-prefix continuation

๐Ÿ“„ License

Licensed under the Apache License 2.0.

โญ If Agentic API saves you from writing one more client-side tool loop, star the repo! โญ