Awesome Scientific LLM Benchmarks

Benchmarks for evaluating large language models on scientific reasoning and discovery — across mathematics, physics & astronomy, chemistry, materials science, biology, and agentic science.

Listed alphabetically within each domain.

Contents

• General / Multi-domain Science
• Mathematics
• Physics & Astronomy
• Chemistry
• Materials Science
• Biology & Life Sciences
• Agentic Science & AI Research

General / Multi-domain Science

Cross-disciplinary STEM reasoning benchmarks; a few are broad exams where science is a major subset.

Benchmark	Org	Year	Paper	Code	Stars	Description
AGIEval	Microsoft	2023	paper	code		Human-centric benchmark from standardized exams like Gaokao, SAT, LSAT, and math competitions.
ARB	DuckAI / Georgia Tech	2023	paper	code		Advanced problems across mathematics, physics, biology, chemistry, and law, including symbolic and proof items graded by rubric.
ARC (AI2 Reasoning Challenge)	Allen AI (AI2)	2018	paper	code		7,787 grade-school science exam multiple-choice questions split into Easy and Challenge sets.
C-Eval	SJTU / HKUST	2023	paper	code		13,948 Chinese multiple-choice questions across 52 disciplines and four difficulty levels.
CharXiv	Princeton University / University of Wisconsin-Madison / HKU	2024	paper	code		2,323 expert-curated charts from scientific papers with descriptive and reasoning questions for realistic multimodal chart understanding.
CURIE	Google	2025	paper	code		580 expert-curated problems drawn from 429 research documents across ten long-context tasks in materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins.
EMMA	CUHK / Microsoft	2025	paper	code		2,788 problems demanding cross-modal reasoning across mathematics, physics, chemistry, and coding for multimodal models.
FrontierScience	OpenAI	2026	paper	—	—	Expert-level science benchmark with an olympiad track (IPhO/IChO/IBO-level) and a rubric-graded PhD-level research track across physics, chemistry, and biology.
GAOKAO-Bench	Fudan University	2023	paper	code		2,811 Chinese college-entrance-exam questions across subjects for LLM evaluation.
GPQA	NYU / Anthropic / Cohere	2023	paper	code		448 expert-written graduate-level biology, physics, and chemistry questions, including the 198-item GPQA Diamond subset used in frontier-model evaluations.
HLE-Diamond	CAIS / Scale AI	2026	paper	—	—	Refined 1,000-question subset of Humanity's Last Exam, split evenly between expert knowledge and reasoning across academic subjects including science and mathematics.
Humanity's Last Exam	CAIS / Scale AI	2025	paper	code		2,500 expert questions across 100+ subjects at the frontier of human academic knowledge.
JEEBench	IIT Delhi	2023	paper	code		515 challenging IIT-JEE Advanced physics, chemistry, and math problem-solving questions.
MMLU-Pro	University of Waterloo (TIGER-Lab)	2024	paper	code		12K reasoning-focused ten-option questions across 14 academic and STEM domains.
MMMU	IN.AI / University of Waterloo / OSU	2023	paper	code		11.5K college-level multimodal questions across six disciplines and 30 subjects.
MMMU-Pro	CMU / University of Waterloo / OSU	2024	paper	—	—	3,460 college-level multimodal questions in standard and vision-only settings, filtered to require visual evidence and expanded to ten answer options.
MMSci	UC Santa Barbara et al.	2024	paper	code		Figure-captioning and multiple-choice tasks built from open-access Nature Communications articles spanning 72 scientific disciplines.
OlympiadBench	Tsinghua University / OpenBMB	2024	paper	code		8,476 olympiad-level bilingual multimodal math and physics problems with expert step annotations.
OlympicArena	SJTU (GAIR)	2024	paper	code		11,163 olympiad-level problems across seven disciplines for multi-discipline cognitive reasoning.
OpenBookQA	Allen AI (AI2)	2018	paper	code		Elementary science multiple-choice questions requiring core facts plus broad common-sense knowledge.
PaperMind	University of Illinois Urbana-Champaign	2026	paper	code		Multimodal benchmark evaluating agent-oriented reasoning and critique over real research papers across seven domains via grounding, experimental interpretation, cross-source evidence, and critical-assessment tasks.
QASC	Allen AI (AI2)	2020	paper	code		9,980 grade-school science questions requiring retrieval and composition of two facts.
ScholarQABench	University of Washington / Allen AI (AI2)	2024	paper	code		2,967 expert-written literature-search queries with 208 long-form expert answers across computer science, physics, neuroscience, and biomedicine, scored on citation-supported synthesis.
SciArena	Yale NLP / Allen AI (AI2)	2025	paper	code		Open platform where researchers vote head-to-head on model answers to literature-grounded science questions, paired with SciArena-Eval for meta-evaluating automated judges.
SciAssess	DP Technology (deepmodeling)	2024	paper	code		Scientific literature analysis over biology, chemistry, materials, and medicine at three levels.
SciBench	UCLA	2023	paper	code		692 open-ended college-level chemistry, physics, and math problems requiring multi-step reasoning.
ScienceQA	UCLA / Allen AI / ASU	2022	paper	code		~21K multimodal multiple-choice science questions with lecture and chain-of-thought explanations.
SciEval	Fudan University (OpenDFM)	2023	paper	code		~18K multi-level questions testing scientific knowledge across chemistry, physics, and biology.
SciFact	Allen AI (AI2)	2020	paper	code		1,409 expert-written scientific claims verified against research abstracts, with rationales.
SciKnowEval	Zhejiang University (HICAI)	2024	paper	code		Tens of thousands of problems across five cognitive levels in biology, chemistry, physics, and materials.
SciQ	Allen AI (AI2)	2017	paper	—	—	13,679 crowdsourced science multiple-choice questions across physics, chemistry, and biology with evidence.
SDE	Cornell / Princeton / Stanford / MIT / Toronto	2025	paper	code		Scenario-grounded scientific-discovery benchmark of 43 scenarios and 1,125 questions across biology, chemistry, materials, and physics, plus 8 project-level hypothesis, experiment-design, and interpretation tasks.
SuperGPQA	ByteDance Seed / M-A-P	2025	paper	code		26,529 graduate-level questions spanning 285 disciplines, including under-evaluated long-tail fields.
TheoremQA	University of Waterloo (TIGER-Lab)	2023	paper	code		800 questions applying 350+ theorems across math, physics, EE/CS, and finance.
Xiezhi	Fudan University	2023	paper	code		249,587 questions spanning 516 disciplines across 13 categories, continuously updated.

Back to top

Mathematics

Arithmetic, competition, olympiad, and frontier / formal-proof mathematics.

Benchmark	Org	Year	Paper	Code	Stars	Description
CHAMP	MIT	2024	paper	code		270 high-school competition problems annotated with concepts and problem-specific hints.
CombiBench	Moonshot AI / Numina	2025	paper	code		100 competition combinatorics problems formalized in Lean4, a domain underrepresented by existing formal benchmarks.
FIMO	Peking University / Huawei	2023	paper	code		149 IMO-shortlist problems formalized in Lean with informal statements for olympiad-level theorem proving.
FormalMATH	SphereLab / M-A-P	2025	paper	code		5,560 formally verified Lean4 statements from olympiad to undergraduate across algebra, calculus, and number theory.
FrontierMath	Epoch AI	2024	paper	—	—	Hundreds of unpublished expert-crafted research-level math problems resistant to guessing.
FrontierMath Erdős	Epoch AI / University of Manchester	2026	paper	—	—	68 Erdős conjectures open as of August 2026, curated for mathematical significance and difficulty, requiring autonomous Lean proofs or disproofs under a fixed budget.
FrontierMath: Open Problems	Epoch AI	2026	paper	—	—	Open research mathematics problems that resisted professional mathematicians, with public problem statements and paid-access programmatic verifiers for proposed solutions.
GSM-Symbolic	Apple	2024	paper	code		Symbolic templates over GSM8K that regenerate each problem with new names, values, and extra clauses to test whether grade-school math accuracy survives superficial perturbation.
GSM8K	OpenAI	2021	paper	code		8.5K grade-school arithmetic word problems requiring multi-step reasoning.
HARDMath	Harvard	2024	paper	code		Graduate applied-math problems requiring asymptotic and approximation methods where leading LLMs score below 45%.
HARP	UCL	2024	paper	code		US competition math problems (AMC/AIME/USAMO) with human-written ground-truth solutions.
IMO-Bench	Google DeepMind	2025	paper	code		Olympiad-level suite testing final answers, proof-writing, grading, and Lean formal proofs, vetted by IMO medalists.
Lean Workbook	Shanghai AI Lab	2024	paper	—	—	About 57K formal-informal Lean4 problem pairs auto-formalized from competition math.
LeanDojo	Caltech / NVIDIA	2023	paper	code		98,734 theorems and proofs from Lean mathlib with premise annotations for retrieval-augmented theorem proving.
MATH	UC Berkeley	2021	paper	code		12,500 competition mathematics problems (AMC/AIME level) with step-by-step solutions.
MATH-500	OpenAI	2023	paper	code		500-problem held-out subset of MATH widely used for LLM evaluation.
MATH-Vision (MATH-V)	CUHK MMLab / Shanghai AI Lab	2024	paper	code		3,040 real competition problems with visual context, spanning 16 mathematical disciplines and five difficulty levels.
MathArena	ETH Zurich / INSAIT	2025	paper	code		Evaluates models only on competitions held after their release date to rule out contamination, covering 162 problems from seven competitions plus human-graded IMO proof writing.
MathBench	Shanghai AI Laboratory	2024	paper	code		Hierarchical bilingual benchmark spanning arithmetic to college-level theory and application.
MATHCHECK	XJTLU / HKUST	2024	paper	code		Checklist benchmark testing task generalization and reasoning robustness beyond end-to-end answer accuracy.
MathOdyssey	NetMind.AI	2024	paper	code		387 expert-crafted high-school, university, and olympiad-level math problems.
MathQA	UW / Allen AI	2019	paper	code		37K multiple-choice math word problems annotated with executable operation programs.
MathVerse	CUHK MMLab / Shanghai AI Lab	2024	paper	code		2,612 visual math problems in six diagram/text variants probing whether MLLMs truly interpret figures.
MathVista	UCLA / University of Washington / Microsoft	2023	paper	code		Mathematical reasoning benchmark combining visual contexts (figures, charts, diagrams) with problems.
miniF2F	OpenAI	2021	paper	code		488 olympiad-level (AMC/AIME/IMO) problems formalized across multiple proof assistants.
Omni-MATH	Peking University	2024	paper	code		4,428 olympiad-level competition problems across 33+ subdomains and difficulty tiers.
ProofBench	Vals AI	2026	paper	—	—	FormalProofBench evaluates advanced undergraduate and graduate mathematics through Lean 4 proof generation, with a 100-problem private test split and iterative tool feedback.
ProofNet	Yale University	2023	paper	code		371 undergraduate theorems for autoformalization and formal proving in Lean.
Putnam-AXIOM	Stanford	2025	paper	—	—	Putnam competition problems plus programmatically perturbed variations giving contamination-resistant unseen instances.
PutnamBench	UT Austin	2024	paper	code		1,600+ Putnam competition problems formalized in Lean, Isabelle, and Coq.
Riemann-bench	Surge AI	2026	paper	—	—	25 private expert-authored research mathematics problems with independently verified, programmatically checkable answers, evaluated with access to coding tools and search.
We-Math	BUPT / Tencent	2024	paper	code		6.5K visual math problems decomposed into 67 hierarchical knowledge concepts to diagnose reasoning versus memorization.

Back to top

Physics & Astronomy

Physics olympiad, graduate physics, computational physics, and astronomy.

Benchmark	Org	Year	Paper	Code	Stars	Description
ABench-Physics	Zhejiang University / Ant Group	2025	paper	—	—	500 hard static plus dynamic-variant physics problems testing reasoning and generalization robustness.
Astro-QA	ACMIS Lab	2025	paper	code		About 2,700 bilingual astronomy questions across six types spanning astrophysics, celestial mechanics, and astrometry.
AstroMLab 1	AstroMLab Collaboration	2024	paper	—	—	4,425 astronomy multiple-choice questions from Annual Reviews testing LLM astronomical knowledge.
AstroVisBench	NSF-Simons CosmicAI Institute	2025	paper	code		Evaluates LLMs on end-to-end astronomy computing workflows and scientific result visualization.
AtmosSci-Bench	HKUST	2025	paper	code		Atmospheric-science problems spanning dynamics, physics, hydrology, geophysics, and oceanography via templated questions.
ClimaQA	UC San Diego	2024	paper	code		Graduate-level climate science QA generated from textbooks with climate scientists in the loop: 566 expert-validated Gold questions plus 3,000 synthetic Silver questions in MCQ, freeform, and cloze form.
CMPhysBench	Chinese Academy of Sciences (IOP)	2025	paper	code		520+ graduate condensed-matter physics problems with a partial-credit metric; top models score below 30%.
CritPt	Argonne National Laboratory / UIUC (50+ physicists, 30+ institutions)	2025	paper	code		71 composite research-project challenges and 190 modular checkpoints authored by 50+ active physicists, auto-graded via numerical, symbolic, and code evaluations.
DiscoverPhysics	Princeton / NYU / Flatiron Institute / Polymathic AI	2026	paper	code		22 simulated worlds with non-standard physics where an agent proposes initial conditions, observes noisy N-body trajectories, and submits a natural-language law plus a Python implementation.
FEABench	Google Research / Harvard	2025	paper	code		Finite-element analysis problems solved end-to-end by driving the COMSOL Multiphysics API: 15 manually verified problems (Gold) plus 200 algorithmically parsed tasks (Large).
Gravity-Bench-v1	University of Toronto	2025	paper	code		Agents plan budgeted observations of simulated binary-star systems to discover the underlying gravitational physics.
HiPhO	CUHK / Shanghai AI Lab	2025	paper	code		13 recent physics olympiad exams with step-level grading and human medal-based comparison for (M)LLMs.
LLM-SRBench	Virginia Tech / CMU	2025	paper	code		239 problems across four science domains testing genuine LLM equation discovery over memorized formulas.
NewtonBench	HKUST	2025	paper	code		324 tasks across twelve physics domains testing LLM agents discovering scientific laws via interactive experimentation.
PHYBench	Peking University	2025	paper	code		500 original physics problems from high-school to olympiad level with expression-edit-distance scoring.
PhysGym	KAUST	2025	paper	code		97 simulated interactive physics-discovery problems with four controlled levels of prior knowledge.
PHYSICS	Yale University	2025	paper	code		1,297 expert-annotated university-level physics problems across six core areas with automated evaluation.
PhysReason	Xi'an Jiaotong University	2025	paper	code		1,200 physics problems with step-level automatic scoring for multi-step physics reasoning.
PRL-Bench	Shanghai Jiao Tong University	2026	paper	—	—	End-to-end physics research benchmark built from ~100 recent Physical Review Letters papers across five subfields, each turned into a long-horizon task scored by an LLM-as-judge.
TPBench	University of Wisconsin-Madison	2025	paper	—	—	57 theoretical physics problems in high-energy theory and cosmology, undergraduate to research level.
UGPhysics	HKUST	2025	paper	code		5,520 bilingual undergraduate physics problems across 13 subjects with rule-based judging.

Back to top

Chemistry

Molecular property, reaction, retrosynthesis, safety, and chemical knowledge.

Benchmark	Org	Year	Paper	Code	Stars	Description
ChemBench	LAMALab, University of Jena (Jablonka group)	2024	paper	code		2,700+ curated questions probing LLM chemical knowledge and reasoning against expert chemists.
ChemCoTBench	IDEA Research / PKU / CUHK	2025	paper	code		Step-wise chemical reasoning across molecular understanding, editing, optimization, and reaction prediction.
ChemEval	USTC	2024	paper	code		Multi-level chemistry benchmark spanning 42 tasks across four progressive difficulty levels for LLMs.
ChemLLMBench	University of Notre Dame et al.	2023	paper	code		Eight chemistry tasks including reaction prediction, retrosynthesis, and molecule captioning for LLMs.
ChemSafetyBench	Peking University	2024	paper	—	—	Safety benchmark testing LLM handling of hazardous-chemical property, legality, and synthesis queries.
MaCBench	LAMALab (Jena) / IIT Delhi	2024	paper	code		1,100+ image-question pairs probing multimodal LLM limitations across chemistry and materials workflows.
MolCap-Arena	Genentech / UIUC	2024	paper	code		Battle-based benchmark scoring 20+ LLMs on molecule captions augmenting molecular property prediction.
MoleculeQA	IDEA Research	2024	paper	code		62K QA pairs over 23K molecules evaluating factual accuracy of LLM molecular comprehension.
MolTextQA	UIUC	2024	paper	code		500K QA pairs over 240K PubChem molecules evaluating molecule-structure-to-text understanding and retrieval.
MOOSE-Chem	NTU Singapore / Shanghai AI Lab	2024	paper	code		Tests whether LLMs rediscover unseen chemistry hypotheses from 51 Nature/Science-level papers given background information.
ScholarChemQA	KAUST / Notre Dame	2024	paper	code		40K abstract-derived chemistry research questions exposing LLM limits in comprehending scholarly literature.
SMolInstruct / LlaSMol	OSU NLP Group	2024	paper	code		Large-scale instruction dataset of 14 small-molecule chemistry tasks used to train and evaluate LlaSMol.
TOMG-Bench	HK PolyU / Shanghai AI Lab	2024	paper	code		Evaluates LLMs on text-based open-domain molecule generation across editing, optimization, and customized generation.

Back to top

Materials Science

Crystals, materials property prediction, and materials-science knowledge.

Benchmark	Org	Year	Paper	Code	Stars	Description
ALDbench	Argonne National Laboratory	2024	paper	code		Expert open-ended QA benchmark evaluating LLMs on atomic-layer-deposition synthesis for accuracy and specificity.
AtomWorld	USTC / Shanghai AI Laboratory / UNSW	2025	paper	code		Evaluates LLM spatial reasoning on crystalline materials via ten atomic-structure editing actions across four modeling categories over CIF files with verifiable metrics.
LLM4Mat-Bench	Princeton (Vertaix)	2024	paper	code		Largest benchmark for LLM materials property prediction over 1.9M crystals and 45 properties.
MaScQA	IIT Delhi (M3RG)	2024	paper	code		650 undergraduate-level materials science questions across 14 categories for evaluating LLM knowledge.
MatCha	CUHK-Shenzhen	2025	paper	code		1,500 expert questions across 21 tasks for materials characterization image understanding by multimodal models.
MatSci-NLP	Mila / Universite de Montreal	2023	paper	code		Seven materials-science NLP tasks evaluating language models via a unified text-to-schema approach.
MatSciBench	UCLA	2025	paper	code		College-level materials-science reasoning benchmark of 1,340 problems spanning six fields, including multimodal questions.
MatText	LamaLab (Jena) / Intel Labs	2024	paper	code		Benchmarking framework evaluating language models on crystal property prediction across nine text representations.
MatViX	Duke University	2024	paper	code		Benchmarks vision-language models on multimodal information extraction from visually-rich materials-science articles.
MatVQA	Mila / Universite de Montreal	2025	paper	—	—	1,325 questions testing multimodal models on materials imagery like microscopy and diffraction with multi-step reasoning.

Back to top

Biology & Life Sciences

Genomics, proteins, bioinformatics agents, protocols, and research biology.

Benchmark	Org	Year	Paper	Code	Stars	Description
AssayBench	Genentech	2026	paper	code		Evaluates frontier LLMs and agents on gene-hit ranking for 1,920 public CRISPR screens across five phenotype classes, scored with chance-adjusted nDCG.
BioCoder	Yale University (Gerstein Lab)	2023	paper	code		Fuzz-tested benchmark evaluating LLMs on generating bioinformatics code with cross-file dependencies and domain knowledge.
BioLLMBench	UCLA	2023	paper	—	—	2,160 runs evaluating GPT-4, Bard, and LLaMA across 24 bioinformatics tasks and metrics.
BioMaze	Peking University	2025	paper	code		5.1K real-research pathway problems testing LLM reasoning over biological pathways and perturbations.
BioML-bench	ScienceMachine	2025	paper	code		AI agents build end-to-end biomedical ML pipelines across protein, omics, imaging, and drug tasks.
Biomni	Stanford	2025	paper	code		General biomedical agent with Biomni-Eval: 433 instances across 10 reasoning tasks.
BioMysteryBench	Anthropic	2026	paper	—	—	99 expert-written bioinformatics tasks over raw datasets, judged on the final biological conclusion, not the path.
BioPlanner	FutureHouse / Francis Crick / Oxford	2023	paper	code		BioProt dataset automatically evaluating LLMs on generating biology experimental protocols as executable pseudocode.
BioProBench	Peking University	2025	paper	code		Tests LLMs on biological-protocol QA, step ordering, error correction, generation, and reasoning across 556K instances.
BixBench	FutureHouse	2025	paper	code		Bioinformatics agents tackle 53 real analysis scenarios with ~300 open-answer research questions.
CellVoyager	Stanford	2025	paper	code		CellBench: 76 scRNA-seq studies test agents predicting which analyses the authors performed.
GeneBench-Pro	OpenAI	2026	paper	—	—	129 messy, judgment-intensive computational-biology problems across ten genomics domains on realistic datasets.
GeneGPT	NCBI	2023	paper	code		Teaches LLMs to call NCBI Web APIs; adds GeneHop, evaluated on GeneTuring.
GeneTuring	Columbia University	2023	paper	code		16 genomics tasks, 1,600 questions probing LLM genomic knowledge and reasoning.
Genome-Bench	Princeton / Stanford	2025	paper	—	—	3,000+ genome-engineering multiple-choice questions mined from expert forum discussions for evaluating LLM reasoning.
GenoTEX	UIUC	2024	paper	code		Expert-curated gene-expression tasks: dataset selection, preprocessing, and gene-trait statistical analysis.
HealthBench	OpenAI	2025	paper	—	—	5,000 multi-turn physician-crafted health conversations scored by rubric criteria for LLM performance and safety.
LAB-Bench	FutureHouse	2024	paper	code		2,400+ MCQs across literature, figures, databases, protocols, and DNA/protein sequence tasks.
LABBench2	FutureHouse (Edison Scientific)	2026	paper	code		Successor to LAB-Bench with ~1,900 biology-research tasks in realistic contexts (literature, figures, protocols, databases), giving a sharp difficulty jump over LAB-Bench.
LifeSciBench	OpenAI	2026	paper	—	—	750 expert-authored free-response life-science research tasks across seven biological domains, rubric-graded.
ProteinLMBench	Shanghai Jiao Tong University	2024	paper	code		944 verified MCQs assessing LLM comprehension of protein sequences and descriptions.
VirBench	Anthropic (with NCBI, Broad Institute, Pachter Lab)	2026	paper	—	—	120 curated viral-sequence retrieval queries across ~40 pathogens testing whether LLM agents pull correct ground-truth sequence data from NCBI Virus.

Back to top

Agentic Science & AI Research

LLM agents that write research code, run data analyses, attempt autonomous discovery, and conduct ML/AI research.

Benchmark	Org	Year	Paper	Code	Stars	Description
AAAR-1.0	Penn State et al.	2024	paper	code		Expert-annotated tasks assessing LLMs on equation inference, experiment design, and paper-weakness identification.
AIRS-Bench	Meta FAIR	2026	paper	code		20 paper-derived tasks across language modeling, mathematics, bioinformatics, and time series testing research agents on idea generation, experimentation, analysis, and iterative refinement.
AstaBench	Allen AI (AI2)	2025	paper	code		ICLR 2026 oral suite with 2,400+ problems across eleven benchmarks covering literature, coding, data analysis, and end-to-end scientific research.
BLADE	University of Washington	2024	paper	code		Twelve datasets and research questions evaluating agents' analytical decisions against expert ground truth.
CORE-Bench	Princeton	2024	paper	code		270 tasks from 90 papers testing agents on computationally reproducing published scientific results.
Curie	University of Michigan	2025	paper	code		Framework and 46-question benchmark for rigorous, automated scientific experimentation across four CS domains.
DiscoveryBench	Allen AI (AI2)	2024	paper	code		264 real plus 903 synthetic tasks for data-driven hypothesis discovery across six domains.
DiscoveryWorld	Allen AI (AI2)	2024	paper	code		Simulated environment with 120 tasks requiring full cycles of hypothesis, experiment, and analysis.
DSBench	UT Dallas / Tencent AI Lab	2024	paper	code		540 realistic data-analysis and data-modeling tasks sourced from ModelOff and Kaggle competitions.
EXP-Bench	University of Michigan	2025	paper	—	—	Benchmarks agents on designing, implementing, and analyzing end-to-end AI research experiments from publications.
HeurekaBench	EPFL (MLBio Lab)	2026	paper	code		Builds benchmarks of open-ended research questions grounded in real studies and their code to evaluate end-to-end AI co-scientist agents (instantiated as sc-HeurekaBench in single-cell biology).
InnovatorBench	GAIR-NLP (SJTU)	2025	paper	code		Evaluates agents on end-to-end innovative LLM-research tasks like data construction, loss design, and reward design.
LitQA2	FutureHouse	2024	paper	—	—	248 expert-written multiple-choice questions requiring retrieval of a specific scientific paper and reasoning over its full text, with 199 questions publicly released.
LiveIdeaBench	Renmin University of China	2024	paper	code		Benchmarks scientific idea generation from minimal keyword context across five dimensions: originality, feasibility, fluency, flexibility, and clarity.
LMR-Bench	UT Dallas	2025	paper	code		28 tasks from 23 NLP papers testing whether agents reproduce masked research code, verified by unit tests.
MLAgentBench	Stanford	2023	paper	code		13 ML experimentation tasks where agents read, write, and execute code to improve performance.
MLE-bench	OpenAI	2024	paper	code		75 Kaggle ML-engineering competitions testing agents against human leaderboards.
MLGym	Meta	2025	paper	code		Gym framework with 13 open-ended AI research tasks spanning vision, NLP, RL, and game theory.
MLR-Bench	National University of Singapore	2025	paper	code		201 workshop-derived tasks evaluating agents across idea generation, experimentation, and paper writing.
MLRC-Bench	University of Michigan	2025	paper	code		Benchmarks agents on proposing and coding novel methods for seven recent ML research competition problems.
Paper2Code	KAIST / DeepAuto.ai	2025	paper	code		Multi-agent framework and benchmark generating functional code repositories that reproduce ML research papers.
PaperBench	OpenAI	2025	paper	—	—	Agents replicate 20 ICML 2024 papers from scratch, graded on 8,316 rubric criteria.
RE-Bench	METR	2024	paper	code		Seven open-ended ML research-engineering environments comparing agents against 61 human experts.
ResearchBench	Shanghai AI Lab / NTU	2025	paper	—	—	Benchmarks scientific discovery via inspiration retrieval, hypothesis composition, and ranking across twelve disciplines.
ResearchCodeBench	Stanford University	2025	paper	code		Challenges LLMs to implement novel contributions from recent ML papers by completing TODO code snippets.
RExBench	Boston University	2025	paper	code		Twelve tasks where agents autonomously implement novel research extensions to existing papers' codebases.
SciAgentGym	Fudan NLP Group	2026	paper	code		Agentic science benchmark pairing an environment of 1,780 domain-specific tools across natural-science disciplines with a tiered suite from elementary tool actions to long-horizon workflows.
SciCode	UIUC / Princeton / Argonne National Lab	2024	paper	code		80 real research coding problems across 16 natural-science subfields, split into 338 subproblems.
ScienceAgentBench	Ohio State (OSU-NLP)	2024	paper	code		102 data-driven discovery tasks from 44 peer-reviewed papers, evaluating agents that write Python programs.
ScienceWorld	University of Arizona / Microsoft Research / Allen AI (AI2)	2022	paper	code		Interactive text environment with 30 elementary-science tasks where agents must run the experiment — melt a substance, test conductivity, breed a plant — instead of reciting the answer.
Scientist-Bench	University of Hong Kong	2025	paper	code		Evaluates fully autonomous idea-to-paper research systems across CV, NLP, data mining, and IR against expert papers.
SciReplicate-Bench	King's College London	2025	paper	code		Benchmarks agents on reproducing executable code for algorithms described in 36 recent research papers.
SUPER	Allen AI (AI2)	2024	paper	code		Evaluates agents on setting up and executing tasks from research code repositories.
Terminal-Bench-Science	Stanford University / Laude Institute / Terminal-Bench-Science Team	2026	paper	code		70 expert-authored research workflows in v0.1.0 across life, physical, earth, mathematical, and engineering sciences, executed in terminal environments and graded by verifier tests.

Back to top

Contributing

Found a missing benchmark or an error? Edit only data/benchmarks.yaml and open a PR — the README, website, and examples regenerate automatically. See CONTRIBUTING.md for the schema and inclusion bar.

License

Released under the MIT License. Each entry links to its original paper and code; benchmark metadata is drawn from public sources.