Skip to content
Projects

izam-mohammed/ragrank has 47 GitHub stars and 15 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-17 01:09 UTC. Last push 2026-11-14.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

AI

README.md

Original

still under development ~ the api moves around a bit

ragrank

Documentation | API reference | Quickstart | Discord | PyPI

ragrank scores your RAG pipeline so you don't have to squint at outputs and go "yeah, that looks about right". you hand it questions, retrieved context and responses ~ it hands back numbers you can put in CI.

install

pip install ragrank

that's everything you need to evaluate. extras are only for provider and framework sdks ~ the stuff not everyone wants installed:

pip install "ragrank[litellm]"      # claude, gemini, ollama, ~100 more
pip install "ragrank[openai]"       # the openai judge
pip install "ragrank[langchain]"    # LangchainLLMWrapper
pip install "ragrank[llama-index]"  # LlamaindexLLMWrapper
pip install "ragrank[hf]"           # from_hfdataset()
pip install "ragrank[all]"          # all of them

from source:

git clone https://github.com/Auto-Playground/ragrank.git && cd ragrank
uv sync --group dev

quick start

export OPENAI_API_KEY="..."
from ragrank import evaluate
from ragrank.dataset import from_dict
from ragrank.metric import response_relevancy

data = from_dict({
    "question": "What is the capital of France?",
    "context": ["France is famous for the Eiffel Tower and its food."],
    "response": "The capital of France is Paris.",
})

result = evaluate(data, metrics=[response_relevancy])
print(result)
Response Relevancy: 0.850

result.to_dataframe() if you want a table, result.to_json() if you don't want pandas.

bring your own model

any model, not just openai. pass it once and every metric uses it:

from ragrank.integrations.litellm import LiteLLM

result = evaluate(data, llm=LiteLLM(model="ollama/llama3"))

there are langchain and llamaindex wrappers too if you're already in one of those. or write your own ~ subclass BaseLLM, implement generate_text, done.

no key, no problem

there's a FakeLLM in the box, so you can wire up your whole eval pipeline before spending a cent:

from ragrank.llm import FakeLLM

evaluate(data, llm=FakeLLM(responses=["0.8", "0.3"]))

it also takes a response_fn if you want the answer to depend on the prompt. handy for tests.

metrics

judged ~ these cost an llm call

metric what it's askin'
faithfulness is the model makin' things up
correctness is the answer right, against a reference
response_relevancy does the answer actually answer the question
response_conciseness or does it waffle
context_relevancy did retrieval pull back anything useful
context_utilization did the model bother to read it
context_reliance is the retriever earnin' its keep, or is the model
safety is there anythin' harmful in there

free ~ no llm, no cost, same answer every time

metric needs
hit_rate mrr precision_at_k recall_at_k ndcg mean_average_precision retrieved_ids + reference_ids
exact_match token_f1 rouge_l levenshtein_ratio string_presence reference
semantic_similarity reference + an embedding model
json_valid nothin'
pii_free answered nothin'

if retrieval is broken, hit_rate=0.31 tells you more than any judge's opinion of your context ~ and it's free. start there.

faithfulness is the one most people want. it splits your answer into claims and checks each against the context, so a bad score points at the sentence that caused it:

result.metadata["claims"]
# [{"claim": "The tower is in Paris.",  "supported": 1.0},
#  {"claim": "It was built in 1750.",   "supported": 0.0}]

RAG_TRIAD is the three that between them tell you where it broke, RETRIEVAL_METRICS is the free ranking set, and SAFETY_METRICS is the three that ask whether you can ship it at all:

from ragrank.metric import RAG_TRIAD, RETRIEVAL_METRICS, SAFETY_METRICS
evaluate(data, metrics=RAG_TRIAD)

every metric says what it costs before you run it:

hit_rate.cost_tier       # CostTier.FREE
faithfulness.cost_tier   # CostTier.LLM_HEAVY

point it at your actual pipeline

everythin' above scores responses you already have. hand it a target and it runs your rag for you ~ so the questions stay fixed and the pipeline is the thing that changes:

def my_rag(question: str) -> tuple[str, list[str]]:
    chunks = retriever.search(question)
    return generator(question, chunks), chunks

result = evaluate(questions, target=my_rag, metrics=[faithfulness])

pass a dataset instead of a list and it reuses the questions and references, throws away the stale answers, and you've got a regression test.

is the judge any good

the number a judge gives you is worth exactly as much as its agreement with a person, which almost nobody measures. label thirty rows by hand:

from ragrank.evaluation import align

print(align(result, human_labels, threshold=0.5))
# Faithfulness vs human labels (n=40)
#   spearman: +0.799
#   bias: +0.115        <- generous by a consistent 0.115
#   kappa: +0.694

bias is the one to look at. generous by a constant is a prompt you can fix. correlatin' at 0.1 means the metric isn't measurin' what you think.

rolling your own

three ways, none of which need a class:

from ragrank.metric import metric, LLMJudge, Guidelines

@metric(name="Has citation", threshold=1.0)
def has_citation(response: str) -> bool:
    return "[" in response

tone = LLMJudge(judge_name="Tone", instructions="Is the tone right for support?",
                rubric={"A": 1.0, "B": 0.5, "C": 0.0})

policy = Guidelines(judge_name="No advice", guidelines="Never give medical advice.")

parameters are injected by name from the datanode, so a function asks for what it needs and gets nothin' else. a parameter that isn't a datanode field is rejected when you define it, not three hundred rows into a paid run.

when one judge isn't enough

judges are noisy. three ways to deal with that:

from ragrank.metric import Jury, Pairwise

# ask several, take the median
Jury(judges=[gpt_judge, claude_judge, local_judge])

# skip absolute scores entirely ~ models are better at "which is better"
Pairwise(baseline_field="reference")

# or just ask the same judge repeatedly and look at the spread
evaluate(data, run_config=RunConfig(repetitions=5, reducer="median"))

Pairwise judges every pair twice with the order swapped, because judges favour whatever came first. a verdict that flips is reported as a tie, not a win.

gating in ci

give a metric a threshold and the result knows whether it passed:

strict = response_relevancy.model_copy(update={"threshold": 0.7})
result = evaluate(dataset, metrics=[strict])

assert result.passed, result

running it properly

RunConfig is the one place run behaviour lives:

from ragrank.evaluation import RunConfig

evaluate(dataset, run_config=RunConfig(
    max_workers=8,        # concurrent metric calls
    max_retries=2,        # retries on a failing llm call, with backoff
    cache=True,           # reuse identical prompts between runs
    repetitions=1,        # score each row n times and reduce
    raise_on_error=False, # one bad row shouldn't kill a 5000-row run
))

by default a run finishes even when rows fail. a row the judge fluffs comes back with score=None and an error explainin' why, instead of taking down the whole thing forty minutes and twelve dollars in.

a typo'd option is an error, not a shrug ~ RunConfig(max_worker=8) raises rather than quietly running with the default.

what it cost you

result.usage
# 240 calls, 61,204 tokens (59,880 in / 1,324 out)

result.cost(per_prompt_token=0.15/1e6, per_response_token=0.60/1e6)
# 0.0097752

no price table ships with the library ~ prices change and vary by provider, so you pass the rates you're actually payin'.

caching

judges run at temperature 0 over a dataset that barely changes, so addin' one metric re-asks the other five exactly the same questions:

evaluate(data, run_config=RunConfig(cache=True))

one flag for an on-disk cache that survives between runs. off by default ~ no surprise files appear unless you ask.

run 1 (cold)       5 calls, 1345 tokens
run 2 (warm)       5 calls, 0 tokens
run 3 (+1 metric)  only the new metric billed

is v2 better than v1

from ragrank.evaluation import compare

diff = compare(baseline, candidate)
print(diff)
# Response Relevancy: 0.517 -> 0.900 (+0.383) [significant]
# Faithfulness:       0.812 -> 0.815 (+0.003) [within noise]

assert not diff.regressed

it says when a change is noise, usin' the standard errors the runs already report. 0.003 is not an improvement, and a library that lets you claim it is one isn't helpin'. regressed_rows points at the datapoints that moved.

in your test suite

from ragrank.testing import assert_metric

def test_bot_stays_grounded():
    assert_metric(node, faithfulness, threshold=0.9)

plain assertions, no custom runner. works with pytest, unittest, or anythin' that understands assert, and every pytest flag keeps workin' because nothin' got wrapped.

failures carry the diagnosis the metric already computed, and point at your line rather than at ours:

Faithfulness scored 0.500, below the threshold of 0.900.
  Unsupported claims:
    - It was built in 1750.

installin' ragrank also registers a small pytest plugin. evals are slow and cost money, so it gives you a way to keep them out of the fast run:

pytest -m "not ragrank"                        # every commit
pytest -m ragrank --ragrank-report=evals.html  # nightly, with a report

from the command line

ragrank eval ragrank.yaml
ragrank eval config.json --output result.json --html report.html
ragrank compare before.json after.json
dataset: data.csv
metrics:
  - faithfulness
  - name: token_f1
    threshold: 0.9
run:
  max_workers: 8
  cache: true

exit 0 passed, 1 a threshold failed, 2 your config was wrong. yaml needs pip install pyyaml; a .json config works with nothin' extra.

data in

from ragrank.dataset import from_dict, from_csv, from_dataframe, from_hfdataset, ColumnMap

from_csv("evals.csv", column_map=ColumnMap(question="query", response="answer"))

already got output from some other tool? hand it the records it already gives you, no reshapin':

from ragrank.dataset import from_records, from_json, from_jsonl

from_json("outputs.json")
from_records([{"question": "...", "context": ["..."], "response": "..."}])

and back out again ~ to_records(), to_json(), to_jsonl(), to_csv().

a report somebody else will read

result.to_html("report.html")

one file, no assets, nothin' to serve. every row, every score, and what the judge said about it. attach it to the pr and let the reviewer see which rows broke instead of takin' your word for it.

development

make test-offline   # no api key needed
make test           # needs OPENAI_API_KEY
make lint
make format

test-offline runs the whole thing against FakeLLM ~ no network, no spend.

license

apache 2.0. do what you like with it.

contributing

issues and PRs welcome. if somethin' is broken, an issue with the traceback is genuinely useful ~ two of the bugs fixed recently were sitting in the tracker for two years because nobody re-tested them after they got closed.