aurelio-labs/semantic-router

Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow LLM generations to make tool-use decisions, it decides in milliseconds: it compares the semantic meaning of a request against your routes in vector space, or asks a small decision model in a single fast call. Every decision comes with a confidence you can calibrate into a real probability.

Read the Docs

---

Versions

Semantic Router has two release lines. The 1.x line (main branch) rebuilds the routing layer around calibrated confidence, decision models and composable routers. The 0.x line (v0 branch) finishes at 0.3.0 and receives bug fixes only.

pip install -qU "semantic-router>=1,<2"   # 1.x
pip install -qU "semantic-router<1"       # 0.x

Upgrading from 0.x? The migration guide covers every change, with before and after code. The 0.x docs stay on the versions page.

---

Quickstart

This quickstart runs entirely on your machine, with a small local embedding model and no API keys:

pip install -qU "semantic-router[fastembed]"

We begin by defining a set of Route objects. These are the decisions the router can make. Let's try two simple routes for now, one for talk on politics and another for chitchat:

from semantic_router import Route

# we could use this as a guide for our chatbot to avoid political conversations
politics = Route(
    name="politics",
    utterances=[
        "isn't politics the best thing ever",
        "why don't you tell me about your political opinions",
        "don't you just love the president",
        "they're going to destroy this country!",
        "they will save the country!",
    ],
)

# this could be used as an indicator to our chatbot to switch to a more
# conversational prompt
chitchat = Route(
    name="chitchat",
    utterances=[
        "how's the weather today?",
        "how are things going?",
        "lovely weather today",
        "the weather is horrendous",
        "let's go to the chippy",
    ],
)

# we place both of our decisions together into single list
routes = [politics, chitchat]

Next we pick an encoder, the model that turns text into vectors. We use a local FastEmbed model here; OpenAI, Cohere, Hugging Face and many more work the same way.

from semantic_router.encoders import FastEmbedEncoder

encoder = FastEmbedEncoder(name="BAAI/bge-small-en-v1.5")

With our routes and encoder defined, we create an IndexedRouter. It embeds every utterance once and matches each query against them.

from semantic_router import IndexedRouter

router = IndexedRouter(encoder=encoder, routes=routes)

Now we can make super fast decisions based on user queries:

router("don't you love politics?").name

[Out]: 'politics'

Correct decision, let's try another:

router("how's the weather today?").name

[Out]: 'chitchat'

We get both decisions correct! By default the best route always wins, so to let the router say "none of these" we calibrate it on a few labelled queries it hasn't seen (None means no route should match) and set a confidence threshold:

router.fit(
    X=[
        "who should I vote for in the election?", "what do you think of the prime minister?", "is the government doing a good job?",
        "tell me your opinion on the new tax law", "are you left wing or right wing?", "the opposition party is a joke",
        "nice day out, isn't it?", "it's freezing outside", "how was your weekend?",
        "fancy grabbing some fish and chips?", "what a lovely sunny afternoon", "how have you been?",
        "how do I reset my password?", "what's the capital of australia?", "write me a python function to sort a list",
        "what is the boiling point of water?", "recommend a good sci-fi book", "how many legs does a spider have?",
    ],
    y=["politics"] * 6 + ["chitchat"] * 6 + [None] * 6,
)
router.confidence_threshold = 0.5  # now a probability: act only when it's more likely right than wrong

router("I'm interested in learning about llama 2").name

[Out]:

No route cleared the bar, so the router returned None!

Beyond embeddings

• DecisionRouter asks a decision model (TypeSafe's Jev, Laya or Von) which route applies, and can answer several yes/no, choice and score questions about one message in a single call. New to decision models? Read Understanding Jev or watch the video.
• Chain answers easy queries from a fast router and falls through to a slower one only when it isn't sure. Merge averages several routers.
• Hybrid search: give IndexedRouter a sparse_encoder to weigh exact keywords alongside meaning.
• Production indexes: keep your routes in Pinecone, Qdrant or Postgres.
• Multi-modal: route on images as well as text.

---

📚 Resources

Docs

Notebook	Description
Introduction	Introduction to Semantic Router and the IndexedRouter
Save/Load Routers	Save any router to YAML or JSON and load it back, calibration included
LangChain Integration	How to integrate Semantic Router with LangChain Agents
Local Execution	Fully local routing with FastEmbed, no API keys
Calibration	Turn raw scores into probabilities with fit(), and set thresholds that mean what they say
Multi-Modal Routes	Using multi-modal routes to identify Shrek vs. not-Shrek pictures
Healthcare Administrative Routing	Fully local routing, calibration, and evaluation across seven synthetic healthcare administrative workflows

Online Course

Community

• Dimitrios Manias, Ali Chouman, Abdallah Shami, Semantic Routing for Enhanced Performance of LLM-Assisted Intent-Based 5G Core Network Management and Orchestration, IEEE GlobeCom 2024
• Julian Horsey, Semantic Router superfast decision layer for LLMs and AI agents, Geeky Gadgets
• azhar, Beyond Basic Chatbots: How Semantic Router is Changing the Game, AI Insights @ Medium
• Daniel Avila, Semantic Router: Enhancing Control in LLM Conversations, CodeGPT @ Medium
• Yogendra Sisodia, Stop Chat-GPT From Going Rogue In Production With Semantic Router, Medium
• Aniket Hingane, LLM Apps: Why you Must Know Semantic Router in 2024: Part 1, Medium
• Adrien Sales, 🔀 Semantic Router w. ollama/gemma2 : real life 10ms hotline challenge 🤯
• Adrien Sales, Kaggle Notebook 🔀 Semantic Router: ollama/ gemma2:9b hotline