Quail

An open-source inference engine for AI functions in SQL
from the Full Stack Data Lab at CMU

Quail (QUery-Aware Inference Layer) runs LLMs over large datasets, from SQL, on your own GPUs. Quail's secret sauce is to optimize LLM inference and memory management for each query as a whole. This makes Quail lightning fast: 1.8x faster on average than well-tuned vLLM baselines, and up to 14x faster on queries with AI joins (details)!

SELECT r.id,
       AI.EXTRACT(r.body, {'actor': 'Which actor does the reviewer name?'}) AS review
FROM reviews r
WHERE AI.IF(PROMPT('Does this review discuss the ending?\n\n{0}', r.body))

Documentation | Quickstart | Blog | QUAIL-B

Install

Install quail-engine from PyPI. In Python, import quail.

uv pip install quail-engine

You need Python 3.12 and a CUDA GPU.

Example

import pyarrow as pa
import quail

reviews = pa.table({
    "id": ["r1", "r2", "r3"],
    "body": [
        "A beautiful film with outstanding performances.",
        "Terrible pacing and a nonsensical plot.",
        "The cinematography was stunning, though the story dragged.",
    ],
})

config = quail.EngineConfig(model="qwen3-4b-fp8", device="h100-sxm")
with quail.Session(config=config) as session:
    session.register("reviews", quail.DocumentProvider.from_table(reviews, id_col="id"))
    result = session.sql("""
        SELECT r.id
        FROM reviews r
        WHERE AI.IF(PROMPT(
            'Does this review mention a positive aspect of the movie?\n\n{0}',
            r.body))
    """, dialect="bq").collect()
    print(result)

Quail Server

You can use Quail Server to run queries on the machine with your GPU. Start the HTTP server and set endpoint to its address. Your queries continue if you disconnect. submit() returns after the server saves your query. Use get_run(query_id) to reconnect later.

uv pip install "quail-engine[server]"
quail-server

import pyarrow as pa
import quail

reviews = pa.table({
    "id": ["r1", "r2"],
    "body": ["The ending was excellent.", "I liked the soundtrack."],
})
sql = """
    SELECT r.id
    FROM reviews r
    WHERE AI.IF(PROMPT('Does this discuss the ending? {0}', r.body))
"""
config = quail.EngineConfig(model="qwen3-4b-fp8", device="h100-sxm")

with quail.Session(
    config=config,
    endpoint="http://127.0.0.1:8642",
) as session:
    session.register(
        "reviews",
        quail.DocumentProvider.from_table(reviews, id_col="id"),
    )
    run = session.sql(sql, dialect="bq").submit()
    for status in run.watch():
        print(status.phase["message"])
    table = run.result().collect()
    print(table)

See Quail Server.

Supported functions

You can use filters and joins evaluated by a language model, EXISTS / NOT EXISTS, numeric scores (AI.SCORE), classification (AI.CLASSIFY), extracting text that answers a question (AI.EXTRACT), and writing a text answer for each document (AI.MAP).

You can write queries in two SQL dialects for AI functions: Snowflake AI_FILTER and BigQuery AI.IF.

Supported models and GPUs

Quail runs Qwen3, Qwen3.5, Qwen3.6, Qwen3.8, LFM2, d1, DiffusionGemma, and Decision 2.0 Kai models on NVIDIA H100 SXM, B200, L40S, and RTX PRO 6000 Blackwell GPUs, with 1, 2, 4, or 8 GPUs per query. See Supported models and GPUs for each model's AI functions and GPU limits.

Development

uv run ruff check quail bench calibrate tests experiments tools
uv run python tools/check_long_strings.py
uv run python tools/check_names.py
uv run vulture
uv run lint-imports
uv run pytest -q

Contributing

See the contributing guide for how to propose and submit changes.