izam-mohammed/ragrank
izam-mohammed/ragrank has 47 GitHub stars and 15 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-17 01:09 UTC. Last push 2026-11-14.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.
AIREADME.md
Originalstill under development ~ the api moves around a bit
Documentation | API reference | Quickstart | Discord | PyPI
ragrank scores your RAG pipeline so you don't have to squint at outputs and go "yeah, that looks about right". you hand it questions, retrieved context and responses ~ it hands back numbers you can put in CI.
install
pip install ragrank
that's everything you need to evaluate. extras are only for provider and framework sdks ~ the stuff not everyone wants installed:
pip install "ragrank[litellm]" # claude, gemini, ollama, ~100 more
pip install "ragrank[openai]" # the openai judge
pip install "ragrank[langchain]" # LangchainLLMWrapper
pip install "ragrank[llama-index]" # LlamaindexLLMWrapper
pip install "ragrank[hf]" # from_hfdataset()
pip install "ragrank[all]" # all of them
from source:
git clone https://github.com/Auto-Playground/ragrank.git && cd ragrank
uv sync --group dev
quick start
export OPENAI_API_KEY="..."
from ragrank import evaluate
from ragrank.dataset import from_dict
from ragrank.metric import response_relevancy
data = from_dict({
"question": "What is the capital of France?",
"context": ["France is famous for the Eiffel Tower and its food."],
"response": "The capital of France is Paris.",
})
result = evaluate(data, metrics=[response_relevancy])
print(result)
Response Relevancy: 0.850
result.to_dataframe() if you want a table, result.to_json() if you don't want pandas.
bring your own model
any model, not just openai. pass it once and every metric uses it:
from ragrank.integrations.litellm import LiteLLM
result = evaluate(data, llm=LiteLLM(model="ollama/llama3"))
there are langchain and llamaindex wrappers too if you're already in one
of those. or write your own ~ subclass BaseLLM, implement
generate_text, done.
no key, no problem
there's a FakeLLM in the box, so you can wire up your whole eval pipeline before spending a
cent:
from ragrank.llm import FakeLLM
evaluate(data, llm=FakeLLM(responses=["0.8", "0.3"]))
it also takes a response_fn if you want the answer to depend on the prompt. handy for tests.
metrics
judged ~ these cost an llm call
| metric | what it's askin' |
|---|---|
faithfulness |
is the model makin' things up |
correctness |
is the answer right, against a reference |
response_relevancy |
does the answer actually answer the question |
response_conciseness |
or does it waffle |
context_relevancy |
did retrieval pull back anything useful |
context_utilization |
did the model bother to read it |
context_reliance |
is the retriever earnin' its keep, or is the model |
safety |
is there anythin' harmful in there |
free ~ no llm, no cost, same answer every time
| metric | needs |
|---|---|
hit_rate mrr precision_at_k recall_at_k ndcg mean_average_precision |
retrieved_ids + reference_ids |
exact_match token_f1 rouge_l levenshtein_ratio string_presence |
reference |
semantic_similarity |
reference + an embedding model |
json_valid |
nothin' |
pii_free answered |
nothin' |
if retrieval is broken, hit_rate=0.31 tells you more than any judge's
opinion of your context ~ and it's free. start there.
faithfulness is the one most people want. it splits your answer into
claims and checks each against the context, so a bad score points at the
sentence that caused it:
result.metadata["claims"]
# [{"claim": "The tower is in Paris.", "supported": 1.0},
# {"claim": "It was built in 1750.", "supported": 0.0}]
RAG_TRIAD is the three that between them tell you where it broke,
RETRIEVAL_METRICS is the free ranking set, and SAFETY_METRICS is the
three that ask whether you can ship it at all:
from ragrank.metric import RAG_TRIAD, RETRIEVAL_METRICS, SAFETY_METRICS
evaluate(data, metrics=RAG_TRIAD)
every metric says what it costs before you run it:
hit_rate.cost_tier # CostTier.FREE
faithfulness.cost_tier # CostTier.LLM_HEAVY
point it at your actual pipeline
everythin' above scores responses you already have. hand it a target
and it runs your rag for you ~ so the questions stay fixed and the
pipeline is the thing that changes:
def my_rag(question: str) -> tuple[str, list[str]]:
chunks = retriever.search(question)
return generator(question, chunks), chunks
result = evaluate(questions, target=my_rag, metrics=[faithfulness])
pass a dataset instead of a list and it reuses the questions and references, throws away the stale answers, and you've got a regression test.
is the judge any good
the number a judge gives you is worth exactly as much as its agreement with a person, which almost nobody measures. label thirty rows by hand:
from ragrank.evaluation import align
print(align(result, human_labels, threshold=0.5))
# Faithfulness vs human labels (n=40)
# spearman: +0.799
# bias: +0.115 <- generous by a consistent 0.115
# kappa: +0.694
bias is the one to look at. generous by a constant is a prompt you can
fix. correlatin' at 0.1 means the metric isn't measurin' what you think.
rolling your own
three ways, none of which need a class:
from ragrank.metric import metric, LLMJudge, Guidelines
@metric(name="Has citation", threshold=1.0)
def has_citation(response: str) -> bool:
return "[" in response
tone = LLMJudge(judge_name="Tone", instructions="Is the tone right for support?",
rubric={"A": 1.0, "B": 0.5, "C": 0.0})
policy = Guidelines(judge_name="No advice", guidelines="Never give medical advice.")
parameters are injected by name from the datanode, so a function asks for what it needs and gets nothin' else. a parameter that isn't a datanode field is rejected when you define it, not three hundred rows into a paid run.
when one judge isn't enough
judges are noisy. three ways to deal with that:
from ragrank.metric import Jury, Pairwise
# ask several, take the median
Jury(judges=[gpt_judge, claude_judge, local_judge])
# skip absolute scores entirely ~ models are better at "which is better"
Pairwise(baseline_field="reference")
# or just ask the same judge repeatedly and look at the spread
evaluate(data, run_config=RunConfig(repetitions=5, reducer="median"))
Pairwise judges every pair twice with the order swapped, because judges
favour whatever came first. a verdict that flips is reported as a tie,
not a win.
gating in ci
give a metric a threshold and the result knows whether it passed:
strict = response_relevancy.model_copy(update={"threshold": 0.7})
result = evaluate(dataset, metrics=[strict])
assert result.passed, result
running it properly
RunConfig is the one place run behaviour lives:
from ragrank.evaluation import RunConfig
evaluate(dataset, run_config=RunConfig(
max_workers=8, # concurrent metric calls
max_retries=2, # retries on a failing llm call, with backoff
cache=True, # reuse identical prompts between runs
repetitions=1, # score each row n times and reduce
raise_on_error=False, # one bad row shouldn't kill a 5000-row run
))
by default a run finishes even when rows fail. a row the judge fluffs
comes back with score=None and an error explainin' why, instead of
taking down the whole thing forty minutes and twelve dollars in.
a typo'd option is an error, not a shrug ~ RunConfig(max_worker=8)
raises rather than quietly running with the default.
what it cost you
result.usage
# 240 calls, 61,204 tokens (59,880 in / 1,324 out)
result.cost(per_prompt_token=0.15/1e6, per_response_token=0.60/1e6)
# 0.0097752
no price table ships with the library ~ prices change and vary by provider, so you pass the rates you're actually payin'.
caching
judges run at temperature 0 over a dataset that barely changes, so addin' one metric re-asks the other five exactly the same questions:
evaluate(data, run_config=RunConfig(cache=True))
one flag for an on-disk cache that survives between runs. off by default ~ no surprise files appear unless you ask.
run 1 (cold) 5 calls, 1345 tokens
run 2 (warm) 5 calls, 0 tokens
run 3 (+1 metric) only the new metric billed
is v2 better than v1
from ragrank.evaluation import compare
diff = compare(baseline, candidate)
print(diff)
# Response Relevancy: 0.517 -> 0.900 (+0.383) [significant]
# Faithfulness: 0.812 -> 0.815 (+0.003) [within noise]
assert not diff.regressed
it says when a change is noise, usin' the standard errors the runs already
report. 0.003 is not an improvement, and a library that lets you claim it
is one isn't helpin'. regressed_rows points at the datapoints that moved.
in your test suite
from ragrank.testing import assert_metric
def test_bot_stays_grounded():
assert_metric(node, faithfulness, threshold=0.9)
plain assertions, no custom runner. works with pytest, unittest, or
anythin' that understands assert, and every pytest flag keeps workin'
because nothin' got wrapped.
failures carry the diagnosis the metric already computed, and point at your line rather than at ours:
Faithfulness scored 0.500, below the threshold of 0.900.
Unsupported claims:
- It was built in 1750.
installin' ragrank also registers a small pytest plugin. evals are slow and cost money, so it gives you a way to keep them out of the fast run:
pytest -m "not ragrank" # every commit
pytest -m ragrank --ragrank-report=evals.html # nightly, with a report
from the command line
ragrank eval ragrank.yaml
ragrank eval config.json --output result.json --html report.html
ragrank compare before.json after.json
dataset: data.csv
metrics:
- faithfulness
- name: token_f1
threshold: 0.9
run:
max_workers: 8
cache: true
exit 0 passed, 1 a threshold failed, 2 your config was wrong. yaml needs
pip install pyyaml; a .json config works with nothin' extra.
data in
from ragrank.dataset import from_dict, from_csv, from_dataframe, from_hfdataset, ColumnMap
from_csv("evals.csv", column_map=ColumnMap(question="query", response="answer"))
already got output from some other tool? hand it the records it already gives you, no reshapin':
from ragrank.dataset import from_records, from_json, from_jsonl
from_json("outputs.json")
from_records([{"question": "...", "context": ["..."], "response": "..."}])
and back out again ~ to_records(), to_json(), to_jsonl(),
to_csv().
a report somebody else will read
result.to_html("report.html")
one file, no assets, nothin' to serve. every row, every score, and what the judge said about it. attach it to the pr and let the reviewer see which rows broke instead of takin' your word for it.
development
make test-offline # no api key needed
make test # needs OPENAI_API_KEY
make lint
make format
test-offline runs the whole thing against FakeLLM ~ no network, no spend.
license
apache 2.0. do what you like with it.
contributing
issues and PRs welcome. if somethin' is broken, an issue with the traceback is genuinely useful ~ two of the bugs fixed recently were sitting in the tracker for two years because nobody re-tested them after they got closed.