Projects
View on GitHub Awesome AI Eval
Vvkmnn/awesome-ai-eval
Vvkmnn/awesome-ai-eval has 127 GitHub stars and 60 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-17 02:14 UTC. Last push 2026-11-01.
Measured star growth
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.
AIREADME.md
OriginalAwesome AI Eval 
Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.
Contents
Contributing
See CONTRIBUTING.md.
Benchmarks
Agents
- Agent Retry Rodeo - Local test range that replays lost acknowledgements and partial failures, then checks an agent's final state.
- AgentBench - LLMs acting as agents across operating system, database, web, and game environments.
- AppWorld - Controllable world of apps and simulated people for benchmarking function calling and interactive coding agents.
- AstaBench - Scientific research tasks covering literature review, replication, and data analysis.
- Berkeley Function Calling Leaderboard - Measures how accurately models call functions and tools.
- BrowseComp - Questions that require persistent web browsing to answer.
- BrowserGym - Gym environment for web agents with access to WebArena, WorkArena, and other benchmarks.
- ClawBench - Browser agents on everyday tasks across live websites, with full action and HTTP traces.
- ColBench - Multi-turn collaborative coding and design tasks with a simulated human partner.
- Context-Bench - Measures how well agents manage context and memory.
- DeepResearch Bench - PhD-level research tasks for scoring deep research agents on report quality and citations.
- DPAI Arena - Benchmark for developer agents across languages and engineering workflows.
- GAIA - General assistant questions requiring web access, tools, and planning.
- MCP-Universe - Benchmark and framework for evaluating agents on real Model Context Protocol servers.
- MetaTool - Tests whether models know when to use a tool and which one to pick.
- MLE-bench - Kaggle competitions measuring agents on machine learning engineering.
- OpenAI Frontier Evals - PaperBench, SWE-Lancer, and EVMbench: paper replication, freelance coding, and smart contract security tasks.
- OSWorld - Computer tasks in real desktop environments for multimodal agents.
- ShopPay Audit Benchmark - Spec-grounded business-logic audit tasks for coding agents.
- SuperCLUE-Agent - Chinese agent evaluation covering tool use, planning, and memory.
- SWE-bench - Real GitHub issues that coding agents must resolve.
- SWE-bench Live - Continuously refreshed issues to limit contamination.
- SWE-bench Pro - Long-horizon coding problems drawn from public, commercial, and held-out repositories.
- tau2-bench - Customer-service simulations testing tool use and policy adherence against a simulated user.
- Terminal-Bench - Continuously updated benchmark of difficult command-line tasks for agents.
- TIB-AK79 - Scores what agent actions changed in a mock environment and checks the agent's claims against the result.
- WebArena - Self-hosted websites for evaluating autonomous web agents.
General
- AGIEval - Human-centric exams spanning admission tests, law, and math.
- ARC-AGI-2 - Abstract reasoning puzzles that are easy for people and hard for models.
- ARC-AGI-3 - Interactive reasoning benchmark of novel game environments in which agents must discover the goals.
- CommonGen-Eval - GPT-4 judged suite for constrained commonsense text generation.
- DyVal - Dynamically generated reasoning tasks with controllable difficulty.
- GDPval - Economically valuable tasks built from the work of professionals in 44 occupations.
- GPQA - Graduate-level science questions written by experts and designed to resist web search.
- Humanity's Last Exam - Expert-written closed-ended questions across academic subjects.
- IFEval - Prompts with verifiable constraints for checking instruction compliance.
- JudgeBench - Benchmark for LLM judges on hard response pairs across knowledge, reasoning, math, and coding.
- LiveBench - Contamination-limited benchmark with regularly refreshed questions and objective scoring.
- LLM-Uncertainty-Bench - Uncertainty-aware scoring across QA, reading comprehension, inference, dialog, and summarization.
- LLMBar - Meta-evaluation of whether LLM judges can spot instruction-following failures.
- MMLU - Multiple-choice questions across academic and professional subjects. The standard knowledge benchmark; complete and no longer updated.
- MMLU-Pro - Harder ten-option extension of MMLU focused on reasoning.
- PertEval - Knowledge-invariant perturbations that expose inflated multiple-choice accuracy.
- RewardBench - Benchmark and leaderboard for evaluating reward models.
- SimpleBench - Everyday reasoning questions where people still outperform frontier models.
- SimpleQA - Short fact-seeking questions with a single indisputable answer, for measuring factuality.
- StructEval - Structured-output generation and conversion across 18 text and renderable formats.
Long-Context
- InfiniteBench - Twelve tasks with average context length beyond 100K tokens.
- LongBench v2 - Questions over contexts from 8K to 2M words in six task categories.
- LongGenBench - Long-form generation benchmark for long-context models.
- LV-Eval - Five length tiers up to 256K words with distraction controls.
- RULER - Synthetic long-context tasks with configurable sequence length.
Multimodal
- MERLIM - 300K+ image-question pairs targeting cross-modal hallucination.
- MME - Perception and cognition across 14 subtasks.
- MMMU-Pro - Harder version of MMMU with expert-level multimodal questions.
- MMT-Bench - 31K+ questions across image, text, video, and point-cloud inputs.
- PaperEdits Agentic Video Benchmark - Exploratory benchmark of agentic and static video processing on long-form editing tasks.
- Video-MME - Video understanding across short, medium, and long clips.
Safety
- AdvBench - Harmful behaviors and strings for measuring jailbreak resistance.
- Anthropic Model-Written Evals - Model-written datasets for testing persona, sycophancy, and AI risk behaviors.
- BBQ - Question sets measuring reliance on social stereotypes.
- Cybench - Capture-the-flag tasks for evaluating the cybersecurity capabilities and risks of agents.
- CyberSecEval - Meta benchmarks measuring cybersecurity risks and defensive capabilities of language models.
- HarmBench - Standardized evaluation of automated red teaming methods and model robustness.
- JailbreakBench - Robustness benchmark with a jailbreak artifact repository and leaderboard.
- SimpleSafetyTests - 100 prompts covering five critical harm areas.
- TruthfulQA - Adversarially written questions that tempt models into common falsehoods.
Specialized
- FinanceBench - Questions over public company filings.
- FinEval - Chinese financial knowledge and reasoning across regulation, accounting, and markets.
- FrontierMath - Epoch AI benchmark of original, exceptionally hard mathematics problems.
- HealthBench - Multi-turn health conversations graded against rubrics written by physicians.
- HumanEval - Hand-written programming problems checked by unit tests. Complete and no longer updated.
- LAiW - Chinese legal benchmark covering retrieval, inference, and case application.
- LiveCodeBench - Contamination-free code evaluation built from newly published competitive programming problems.
- MATH - Competition mathematics problems with step-by-step solutions.
- MBPP - Entry-level Python programming problems with tests.
- MedHELM - Clinician-validated medical tasks scored with an LLM jury.
Leaderboards
- ARC Prize Leaderboard - ARC-AGI scores for frontier models and open submissions.
- Arena - Crowdsourced head-to-head model voting, formerly LMArena.
- CompassRank - OpenCompass rankings across multi-domain suites.
- Council of AI GSPC - Public board of signed AI behaviour measurements across governance and safety axes.
- Epoch AI Benchmarking Hub - Epoch AI results for leading models on challenging benchmarks.
- HELM Leaderboards - Stanford results across HELM scenarios and metrics.
- ImageBench - Text-to-image comparisons with side-by-side results and published methodology.
- MathArena - Evaluation of LLMs on the latest math competitions, run after each contest.
- ModelBenchmark - Public benchmark scores, prices, and context windows gathered per model.
- Open LLM Leaderboard - Hugging Face rankings of open models on IFEval, MMLU-Pro, GPQA, and more.
- Open Medical-LLM Leaderboard - Hugging Face rankings on medical question answering.
- Scale SEAL Leaderboard - Expert-rated rankings for reasoning, coding, and safety.
- Vectara Hallucination Leaderboard - Ranks LLMs by how often they hallucinate when summarizing short documents.
Platforms
Cloud
- Amazon Bedrock Evaluations - Managed scoring of foundation models and RAG pipelines.
- Google Gen AI Evaluation Service - Rubric-based evaluation of models and agents on Google Cloud.
- Microsoft Foundry Evaluations - Built-in evaluators and reports for generative AI apps.
Managed
- Athina AI - Evaluation and monitoring platform with preset evals and VPC deployment.
- Braintrust - Evaluation and observability platform with experiments, datasets, and CI regression tests.
- ChatIntel - No-code toolkit for evaluating and measuring the consistency of AI output.
- Confident AI - DeepEval-backed platform for scheduled eval suites and production monitoring.
- Datadog LLM Observability - LLM traces, quality metrics, and safety signals inside Datadog.
- Deepchecks LLM Evaluation - Managed eval suites with dataset versioning and alerting.
- Firedrill - Simulates AI agents with optional synthetic tools, semantic scoring, exact checks, and retained evidence in customer-owned runners.
- Future AGI - Evaluation, simulation, and optimization platform for multimodal AI systems.
- Galileo - Observability and evaluation platform with built-in metrics and guardrails.
- HoneyHive - Agent observability and evaluation platform with prompt versioning.
- Jetty - Runs agent workloads in sandboxes and scores each run with checks and LLM judges.
- LangSmith - Tracing, datasets, and batch evals with regression gating.
- Lunary - Observability and prompt management platform for LLM apps.
- PromptLayer - Prompt management with version control and evaluation tracking.
- Respan - AI gateway with tracing, monitoring, red teaming, and evaluators that score test sets and production traffic.
- TestMu AI Agent Testing - Tests chat, voice, and phone agents through simulated conversations and real calls, including their tool calls.
- W&B Weave - Evaluation toolkit with dataset versioning, scorers, and comparison dashboards.
Self-Hosted
- Arize Phoenix - OpenTelemetry-native tracing and evaluation for RAG, LLMs, and agents.
- Bifrost - Self-hosted AI gateway with LLM-judge guardrails and evaluation integrations such as Patronus AI.
- Giskard - Testing library that scans LLM apps for vulnerabilities and quality issues.
- Helicone - AI gateway and observability platform with cost tracking and evaluation integrations.
- Langfuse - Engineering platform for tracing, evals, datasets, and prompt management.
- Latitude - Platform for observing agent traces and tracking recurring failures as issues.
- OpenLIT - OpenTelemetry instrumentation for LLM apps with quality metrics and guardrail hooks.
- Opik - Evaluation and observability platform with datasets, scoring jobs, and trace inspection.
- Rhesis - Collaborative testing platform with test generation and multi-turn conversation simulation.
- Zeno - Evaluation UI for slicing failures and comparing models and prompts.
Resources
Guides
- AI Evals for Engineers & PMs - Cohort course by Hamel Husain and Shreya Shankar.
- AI Failure Mode Taxonomy - Catalog of AI and LLM product failure modes with detection and mitigation for each.
- AlignEval - Eugene Yan on building and aligning LLM judges.
- Applied LLMs - Lessons from a year of building with LLMs, with evaluation as a core practice.
- Comparing LLMs on Real-World Retrieval - How models perform on practical retrieval tasks.
- Data Flywheels for LLM Applications - Iterative data improvement for LLM systems.
- Demystifying Evals for AI Agents - Anthropic guide to grader types, consistency metrics, and building an agent eval suite.
- Eval Tools Comparison - Comparison of Braintrust, LangSmith, and Promptfoo.
- Evals for AI Engineers - O'Reilly book on error analysis, evaluation pipelines, and LLM judges.
- EvalScope RAG - Tutorial on evaluating RAG applications with Ragas, from the EvalScope documentation.
- Field Guide to Rapidly Improving AI Products - Error analysis, data viewers, and systematic improvement from 30+ implementations.
- KDD 2025 Tutorial: Evaluation & Benchmarking of LLM Agents - Academic tutorial on agent evaluation methodology.
- LLM Evals FAQ - Answers to common evaluation questions from practitioners.
- LLM Evaluators Survey - Survey of LLM judge use cases and implementation patterns.
- LLM-as-a-Judge Guide - Using LLMs as judges, with calibration advice.
- Mastering LLMs Open Course - Free course covering evals, RAG, and fine-tuning.
- Microsoft Foundry Safety Evaluations Transparency Note - Microsoft documentation of how the safety evaluators work and where they fall short.
- Open Source LLM Tools Comparison - PostHog comparison of open-source observability and evaluation tools.
- OpenAI Cookbook Evals - Notebooks showing how to build custom evals.
- Why AI Evals Are the Hottest New Skill - Interview covering error analysis, axial coding, and eval prompts.
- Your AI Product Needs Evals - Why every AI product needs systematic evaluation.
- ZenML Evaluation Playbook - Guide to adding evaluation gates to pipelines and deployments.
Lists
- Awesome Agent Evals - Annotated papers, posts, talks, tools, and benchmarks for building and evaluating AI agents.
- Awesome Red Teaming LLMs - Papers on red teaming language models, from a survey accepted at TMLR.
- Awesome-LLM-as-a-judge - Papers on using LLMs as judges, from the accompanying survey.
- Awesome-LLM-Eval - Evaluation tools, papers, datasets, and leaderboards, in Chinese and English.
- Awesome-RAG-Evaluation - RAG evaluation metrics, datasets, and leaderboards.
- LLM Evaluation Survey - Papers organized by the survey "A Survey on Evaluation of Large Language Models".
- LLM-Agent-Benchmark-List - Benchmarks for evaluating LLMs and LLM-based agents.
Papers
- Adding Error Bars to Evals - Statistical methods for analyzing and reporting uncertainty in language model evaluations.
- Don't Pass@k - Bayesian framework that reports LLM evaluation results with uncertainty in place of pass@k.
- Establishing Best Practices for Building Rigorous Agentic Benchmarks - Checklist for avoiding task and reward design flaws in agent benchmarks.
- LLM-as-a-Judge Survey - Survey on building reliable LLM judges and mitigating their biases.
- RAG Evaluation Survey - Survey of metrics, judgments, and open problems in evaluating RAG.
- The Leaderboard Illusion - Analysis of systematic issues that distort Chatbot Arena rankings.
- What's Missing in Autonomous Research? - Survey mapping autonomous research systems against their benchmarks and verification methods.
- Who Validates the Validators? - Paper on aligning LLM judges with human criteria through the EvalGen workflow.
Tools
Adversarial
- AgentDojo - Environment for testing prompt-injection attacks and defenses on tool-using agents.
- DeepTeam - Red teaming framework for bias, data leakage, and prompt-injection vulnerabilities.
- Garak - NVIDIA vulnerability scanner that probes LLMs for prompt injection, leakage, and jailbreaks.
- Petri - Auditing agent that explores model behavior through simulated multi-turn scenarios.
- PyRIT - Microsoft toolkit for automating attack strategies against generative AI systems.
Frameworks
- Agent Stack - Libraries for agent reliability, including snapshot tests for tool-call traces, tool-argument validation, and structured output with retry.
- agentest - Offline, deterministic checks of agent OpenTelemetry traces, A2A AgentCards, and MCP tools, with no LLM judge.
- Agentrial - Runs agents repeatedly, computes confidence intervals, and flags regressions in CI.
- agenttrace - Terminal UI for reviewing coding-agent sessions by cost, failures, latency, and diffs.
- AI Eval Forge - Regression harness for LLM and agent outputs using deterministic checks such as regex, token F1, and JSON validity.
- AI Prompt Quality - Static prompt evaluation with a seven-dimension rubric and JSON scoring schema.
- Aleph Alpha Eval Framework - Evaluation framework with built-in benchmarks for reasoning, coding, and safety.
- align - Human-in-the-loop evals for Claude Code that rate individual claims and archive corrections.
- autoevals - Scorers for LLM outputs, from string heuristics to model-graded factuality, in Python and TypeScript.
- Bloom - Agentic framework that generates behavioral evaluations of frontier models from a described behavior.
- ChainForge - Visual environment for comparing prompts and models and scoring responses.
- ColossalEval - Pipeline combining classic metrics with GPT-assisted scoring across public datasets.
- DeepEval - Unit-test style metrics for hallucination, relevance, toxicity, and bias.
- digline - Regression testing for LLM apps against a versioned baseline of approved scores.
- EvalScope - ModelScope framework for evaluating and benchmarking LLMs, vision-language models, and generative models.
- Evidently - Evaluation and monitoring library for ML and LLM systems.
- fact-gate - Deterministic verification of AI-written text against diffs, measured facts, or prose.
- Harbor - Framework for evaluating and optimizing agents and models in containers; the official Terminal-Bench 2.0 harness.
- HELM - Stanford framework for evaluating foundation models across scenarios and metrics.
- Inspect AI - UK AI Security Institute framework for scripted evals with tool calls and model-graded scoring.
- Inspect Evals - Community-maintained collection of evaluations implemented in Inspect AI.
- lighteval - Hugging Face toolkit for evaluating LLMs across many backends and tasks.
- LM Evaluation Harness - Harness for running language models on hundreds of academic benchmarks.
- lmms-eval - Multimodal evaluation toolkit covering text, image, video, and audio tasks.
- MLflow - Evaluation API that logs judge scores alongside experiment tracking runs.
- Narrative Scorer - Scoring library for Chinese autobiographical narratives in reminiscence therapy.
- OpenAI Evals - Framework and registry of evals for LLMs and LLM systems.
- OpenAI simple-evals - Reference implementations of standard benchmarks such as MMLU, GPQA, and HumanEval.
- OpenCompass - Evaluation platform with LLM judge support across knowledge, reasoning, and coding datasets.
- openevals - Evaluators for LLM apps, including LLM judge and structured-output checks.
- OpenJudge - Framework of graders for evaluating agents and chatbots and producing quality rewards.
- Prompt Flow - Flow builder with built-in evaluation flows and CI integration.
- Promptfoo - CLI for testing and red teaming prompts, RAG flows, and agents with regression detection.
- semantix-ai - NLI judges, datasets, and audit trails for checking LLM outputs against regulatory clauses.
- skill-optimizer - Docker workbench and agent skill for running deterministic evals against agent skills.
- TruLens - Feedback functions for evaluating and tracing chains and agents.
- Voice Eval - Regression harness for voice agents using scripted multi-turn scenarios.
Retrieval
- BEIR - Benchmark suite covering dense, sparse, and hybrid retrieval tasks.
- LlamaIndex Evaluation - Modules for scoring retrievers and comparing query engines.
- MTEB - Embedding benchmark measuring retrieval, reranking, and similarity quality.
- Open RAG Eval - Vectara harness with UMBRELA and AutoNuggetizer metrics that need no golden answers.
- Ragas - Metrics for answer quality, context relevance, and faithfulness in RAG pipelines.
- RagCiteCheck - Measures document-level and span-level evidence stability in RAG pipelines across runs.
- RAGTruth - Human-annotated hallucination corpus for RAG answers.
- UltraRAG - MCP-based RAG framework with built-in evaluation workflows.