Skip to content
Projects

yzhao062/auditable has 31 GitHub stars and 2 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 19:32 UTC. Last push 2026-11-15.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Audit any agent decision across its past, present, and future, on one typed graph.

AI

README.md

Original
auditable

auditable

A library of auditing methods for AI agents, with a leaderboard for each task: configuration audits before a run, online monitors while it runs, and failure detection and fault localization after it.

PyPI Python License CI Open In Colab

Leaderboard · Put Your Method on the Board · Tasks · Quick Start · Real Agent Traces · Methods · Install · Adding a Method · Citation

auditable is a library of auditing methods for AI agents with a leaderboard for each task: one command reruns the CatchBench board on your machine, and a method you write gets a rank on the same population and protocol.

Leaderboard

Did the run fail? The board below ranks 17 methods by ROC-AUC on 376 SWE-Gym OpenHands runs, 188 of which failed. One command reruns it on the population and protocol of the CatchBench board:

pip install "auditable[bench]"
python -m auditable.bench detect/swegym
Rank Method Algorithm ROC-AUC
1 run.pygod_gaan GAAN adversarial detector (PyGOD) 0.850
2 run.lr_spline_flat Logistic regression on a cubic spline of run size, a control 0.832
3 run.gsafeguard_lite G-Safeguard-Lite (supervised graph classifier) 0.828
4 run.lr_full Logistic regression on all raw layers (GRADE full) 0.819
5 run.lr_flatdep Logistic regression on size and dependency (GRADE flatdep) 0.804
6 run.guardian_lite GUARDIAN-Lite (attribute-reconstruction autoencoder) 0.767
7 run.pyod_ecod ECOD on run size 0.765
8 run.pygod_conad CONAD contrastive graph autoencoder (PyGOD) 0.750
9 run.lr_flat Logistic regression on run size (GRADE flat), the size baseline 0.663
10 run.pyod_copod COPOD on run size 0.625
11 run.pygod_anomalydae AnomalyDAE dual autoencoder (PyGOD) 0.592
12 run.pyod_lof Local outlier factor on run size 0.584
13 run.pyod_iforest Isolation Forest on run size 0.571
14 run.pygod_dominant DOMINANT graph autoencoder (PyGOD) 0.547
15 run.random Random run scores, a control 0.483
16 run.pyod_knn KNN distance on run size 0.446
17 run.pyod_hbos HBOS on run size 0.319

PyGOD's GAAN, a literature baseline, leads this board at 0.850. Second, at 0.832, is the flexible size control run.lr_spline_flat: a cubic spline of the same four run-size counts the linear size baseline run.lr_flat reads (0.663). Run size alone, fitted without a linear restriction, ranks above G-Safeguard-Lite and above both GRADE dependency regressions (0.819 and 0.804), so this board does not show that dependency structure predicts failure beyond size, the reading GRADE retired on 2026-09-08. Read a new row against both size controls; on tau-bench the order changes, and run.lr_flatdep (0.665) ranks above the spline control (0.631).

The 16 CatchBench rows equal the CatchBench board, which was computed on Linux; the spline control is not a board row, and its reference value is the size (spline) arm of CatchBench's committed size audit. On macOS, run.pyod_lof prints 0.582 instead of 0.584, and --golden reports that as a platform value and not as a mismatch. The first run downloads the data through agentpile; later runs load SWE-Gym from the cache in about 3 seconds, and the full board takes about 70 seconds on a laptop CPU, of which PyGOD GAAN is the slowest row at about 37 seconds. The graph rows need auditable[graph-ad] and the PyG sampling backend (Install); without them those rows are listed as skipped with the extra named, and the rest of the board is still scored.

python -m auditable.bench                                                    # list the tasks
python -m auditable.bench detect/swegym --methods run.lr_flat run.pyod_ecod  # score chosen rows only
python -m auditable.bench detect/swegym --golden                             # exit 1 if a row differs from the CatchBench board
python -m auditable.bench detect/swegym --markdown                           # a Markdown table; --json prints JSON

Put Your Method on the Board

A method is a class with a MethodCard. Pass it as extra= and leaderboard scores it beside the default rows, under the protocol of its kind: a supervised detector gets the mean ROC-AUC over five seeds of stratified 5-fold cross-validation with a fresh estimator per fold, and an unsupervised one gets a single fit_score over the whole population. The snippet below adds a random forest on run size, a deliberately plain method:

from sklearn.ensemble import RandomForestClassifier

from auditable.base import BaseRunDetector, MethodCard
from auditable.bench import leaderboard
from auditable.featurize import grade_layer_matrix


class ForestOnSize(BaseRunDetector):
    card = MethodCard(method_id="run.forest_flat", name="Random forest on run size", family="run",
                      fit_scope="inductive", supervised=True, output_kind="score", fidelity="original")

    def fit(self, runs, y=None):
        self.model_ = RandomForestClassifier(200, random_state=0)
        self.model_.fit(grade_layer_matrix(list(runs), "flat"), y)
        self.decision_scores_ = self.decision_function(runs)
        return self

    def decision_function(self, runs):
        return self.model_.predict_proba(grade_layer_matrix(list(runs), "flat"))[:, 1]


board = leaderboard("detect/swegym", extra=[ForestOnSize])
print(board.rank_of("run.forest_flat"), board["run.forest_flat"].scores)

The last line prints the rank of the new row among all rows and its scores. print(board) shows the full table, with the CatchBench value in brackets beside each row that has one, and board.to_markdown(), board.to_dict() and board.compare_golden() export it. methods=[...] scores only the rows you name, which is quick while you iterate (leaderboard("detect/swegym", methods=["run.lr_flat", ForestOnSize])), and Entry(method, kwargs={...}, label="...", key="...") adds a row for a variant with constructor arguments (give each variant its own key, because a row that keeps the method id is compared with that method's CatchBench value). A method that raises is listed as an error and the rest of the board is still scored. The method contract lists the conventions for the four families.

Tasks

Six tasks ship with 0.4.0. Each fixes a population and a protocol, so every method on a board is scored on the same data in the same way. detect scores the run detectors, localize the step locators, live the online monitors (run detectors also take part, scored on prefixes of the run), and pre the configuration auditors.

Task Question Population Metric
detect/swegym Did the run fail? SWE-Gym OpenHands trajectories, one scaffold: 376 runs, 188 failed ROC-AUC
detect/tau Did the run fail? tau-bench: 660 runs, 363 failed ROC-AUC
localize/whoandwhen Which step was the decisive mistake? Who&When: 126 failed multi-agent runs with a human mistake step top-1, top-3, MRR (mean reciprocal rank)
live/swegym Can the failure be caught early? SWE-Gym runs, read as prefixes of 25, 50, 75 and 100 percent of their steps mean prefix ROC-AUC
live/tau Can the failure be caught early? tau-bench runs, read as prefixes of 25, 50, 75 and 100 percent of their steps mean prefix ROC-AUC
pre/over_privilege Is the agent configuration over-privileged before it runs? 1,187 configurations from six sources F1, precision, recall

Every task id works with the commands above: python -m auditable.bench <task> reruns the board, and leaderboard("<task>", extra=[...]) places a method on it. The LLM judges on localize/whoandwhen and pre/over_privilege are replayed from committed caches, so those boards need no API call.

The CatchBench board shows how the other tasks read. On Who&When localization, the best LLM judges (all-at-once) reach top-1 of about 0.42 to 0.45 against 0.119 for random, and the structural blast rule ties the position prior at 0.159 top-1. The tau-bench detection board is led by run.lr_flatdep and run.lr_full, tied at 0.665, against 0.619 for the size baseline. For over-privilege, the best rule scanner config.owasp_asi_combined reaches F1 0.654, flagging every configuration reaches 0.601, and an LLM judge reaches 0.695.

What Is in the Library

auditable collects methods that audit LLM agents at three points: an agent configuration before it runs (PRE), a run while it streams (LIVE), and a finished run (POST). Methods are grouped into four families, and each family has one interface, in the way PyOD gives outlier detectors one interface. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN, G-Safeguard, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench.

CatchBench is the paper and benchmark whose board the leaderboards reproduce: auditable.bench carries its populations and evaluation protocols, and every method here that appears on its board reproduces its row sample by sample (Reproducing CatchBench). agentpile is the converter that reads public agent-trace datasets into the one record the methods consume (Real Agent Traces).

Quick Start

Install with pip install "auditable[tabular-ad]" (see Install), then:

from sklearn.metrics import roc_auc_score

from auditable.datasets import make_traces
from auditable.methods import get_method

# 100 toy agent runs; each failed run holds one stale read (a step relying on old state)
traces, y, mistakes = make_traces(n_runs=100, contamination=0.5, random_state=0)
train, test = slice(0, 50), slice(50, 100)

# Did the run fail? Fit on labelled runs, then score new ones, as with a PyOD detector.
detector = get_method("run.lr_flatdep")()
detector.fit(traces[train], y[train])
scores = detector.decision_function(traces[test])   # one score per run, higher = more likely failed
print(f"run-level ROC-AUC: {roc_auc_score(y[test], scores):.3f}")

# Which step was the mistake? Rank the steps of each failed run, most suspected first.
locator = get_method("step.dep_anomaly")()
failed = [i for i in range(100) if y[i] == 1]
hits = sum(locator.rank(traces[i])[0] == mistakes[i] for i in failed)
print(f"mistake ranked first in {hits} of {len(failed)} failed runs")

# Can it be caught live? One causal alarm per arriving step (step t sees steps 0..t only).
monitor = get_method("online.span_z")()
alarms = monitor.step_alarm_scores(traces[failed[0]])
print(f"first alarm peak at step {alarms.argmax()}, mistake at step {mistakes[failed[0]]}")

Every method is created from its ID with get_method(id)(), and the methods of one family share one interface, so trying another detector or locator changes one string. examples/example_methods.py adds a configuration audit before the run.

Real Agent Traces

agentpile loads public agent-trace datasets (Who&When, TRAIL, MAST, tau-bench, SWE-Gym and others), trace files (OpenAI-style messages, OTLP JSON, Claude Code sessions, MCP tool lists) and captured runs into one record. from_agentpile turns those records into the traces every method reads, and configs_from_agentpile turns them into the configurations the config methods read. pip install "auditable[agentpile]" installs the conversion; the example below also runs tabular methods, so it needs pip install "auditable[agentpile,tabular-ad]".

import agentpile as ap
import auditable as au
from auditable.datasets import load_traces
from auditable.methods import get_method

# Any run: name the dependency rules, since agentpile infers none when it loads.
runs = list(ap.load("tau_bench", limit=100))
traces = au.from_agentpile(runs, deps=["tool_call_id_v1", "value_flow_v1"])
scores = get_method("run.pyod_ecod")().fit_score(traces)   # one score per run
y = au.run_targets(traces)                                 # 1 failed, 0 succeeded, -1 unknown

# The same shapes as make_traces, on a public dataset; mistakes holds the labeled step (-1 if none).
traces, y, mistakes = load_traces("who_and_when", deps="full_context", subset="Algorithm-Generated")
labeled = [t for t, m in zip(traces, mistakes) if m >= 0]
print(get_method("step.dep_anomaly")().rank(labeled[0])[:3])

# Before a run: an MCP tools/list result plus the task the agent will get.
configs = au.configs_from_agentpile(next(ap.load("tools.json")), task="Summarize the open issues.")
print(get_method("config.owasp_asi_combined")().audit(configs[0]))

deps has no default. It is a list of agentpile rules, "recorded" for the dependencies the record already holds, or "full_context" for every earlier step, and each trace's meta keeps the rules that ran. The default convention="generic" makes one step per message and one per tool call with its result. convention="grade" with deps="grade" instead reproduces the steps, dependencies and graphs of the GRADE converters that CatchBench and the GRADE paper used, for the seven corpora below. benchmarks/agentpile_crosscheck compares that output with GRADE's run by run.

Corpus load_traces arguments Runs in GRADE's output Result
Who&When "who_and_when", convention="grade", deps="grade" 184 equal; the 126 localization labels match
tau-bench "tau_bench", convention="grade", deps="grade", model=[...] 660 equal
tau2-bench "tau2_bench", convention="grade", deps="grade", model=... 278 equal to the frozen rule tau2-frozen-v1, since GRADE's own tau2 output changes with the hash seed
SWE-Gym "swe_gym_openhands", convention="grade", deps="grade" 376 equal
AgentRewardBench "agent_reward_bench", convention="grade", deps="grade" 600 equal on 3 runs; on the other 597, GRADE adds two steps for the page after the last action, which agentpile does not create, and the steps before them are equal
SWE-rebench OpenHands "swe_rebench_openhands", convention="grade", deps="grade" 600 equal
SWE-agent "swe_agent_nebius", convention="grade", deps="grade" 600 equal

These results hold on the dataset revisions agentpile 0.3.0 loads by default. On any other revision, convention="grade" raises an error rather than return steps nobody has compared with GRADE's, and convention="generic" reads the run without that guarantee.

In the generic convention, a user step is a step the record attributes to the user role, which is not always text a person typed (a session's compact summary or a task notification has that role too). Content given as a list of parts (dicts with a string type, possibly mixed with bare strings) keeps its text; an image, a document or any other non-text part becomes a placeholder such as [image] and its other fields are dropped, while any other list stays JSON. meta["report"] counts the replaced parts and accounts for every step of the record.

A run with subagent trajectories (a Claude Code session with subagents, an Agents SDK handoff) raises an error, since this version does not flatten them into one trace. from_agentpile(run.subagents, ...) reads each subagent as a trace of its own, and subagents="skip" reads the top-level trajectory only; on the Claude Code sessions with subagents we measured, skipping left out about half of the steps.

Methods

Configuration rules and the step-by-step and binary-search LLM judges run on the core install, graph detectors need the graph-ad extra, and every other method needs tabular-ad. The tables are generated from the registry: list_methods(family=...) returns each method's card, and method_table(detailed=True) prints the fit scope, fidelity to the source, extras, and the CatchBench row each method reproduces. The method contract defines the four interfaces on one page.

Config: Is the Agent Over-Privileged Before It Runs?

auditor.audit(config) reads an AgentConfig (the declared capabilities and the task text) and returns a finding: flag (True, False, or None to abstain), the excess capabilities, and the rules that fired.

Type Method Algorithm Reference
Rule config.flag_risky_perms Flag risky permission levels CatchBench (2026)
Rule config.owasp_asi_combined Combined OWASP and CWE scanner CatchBench (2026)
Rule config.owasp_excess_functionality Excess functionality CatchBench (2026)
Rule config.owasp_excess_permissions Excess permission level CatchBench (2026)
Rule config.owasp_privilege_escalation Unrequested privilege management CatchBench (2026)
Rule config.sensitive_access Unrequested sensitive access CatchBench (2026)
Rule config.unrequested_high_impact Unrequested high-impact action CatchBench (2026)
LLM judge config.llm_needed LLM judge, needed capabilities CatchBench (2026)

Online: Can the Failure Be Caught While the Run Streams?

monitor.step_alarm_scores(trace) returns one alarm per step, and the alarm at step t reads steps 0 to t only.

Type Method Algorithm Reference
Rule online.dep_count_z Causal z-score of dependency count CatchBench (2026)
Rule online.dep_span Mean dependency span (running) CatchBench (2026)
Rule online.raw_span Raw dependency span CatchBench (2026)
Rule online.span_z Causal z-score of dependency span CatchBench (2026)
Baseline online.random Random alarm scores CatchBench (2026)

Run: Did the Run Fail?

detector.fit(traces, y), then detector.decision_function(traces): one score per run, higher meaning more likely failed. Unsupervised detectors ignore y. Transductive detectors (the PyGOD detectors and GUARDIAN-Lite) fit and score one batch together with detector.fit_score(traces).

Type Method Algorithm Reference
Supervised run.lr_flat Logistic regression on run size (GRADE flat) GRADE (Zhao, 2026)
Supervised run.lr_flatdep Logistic regression on size and dependency (GRADE flatdep) GRADE (Zhao, 2026)
Supervised run.lr_full Logistic regression on all raw layers (GRADE full) GRADE (Zhao, 2026)
Supervised run.lr_spline_flat Logistic regression on a cubic spline of run size (flexible size control) CatchBench (2026)
PyOD detector run.pyod_copod COPOD on run size COPOD (Li et al., 2020)
PyOD detector run.pyod_ecod ECOD on run size ECOD (Li et al., 2022)
PyOD detector run.pyod_hbos HBOS on run size HBOS (Goldstein and Dengel, 2012)
PyOD detector run.pyod_iforest Isolation Forest on run size Isolation Forest (Liu et al., 2008)
PyOD detector run.pyod_knn KNN distance on run size KNN outlier score (Ramaswamy et al., 2000)
PyOD detector run.pyod_lof Local outlier factor on run size LOF (Breunig et al., 2000)
Graph detector run.gsafeguard_lite G-Safeguard-Lite (supervised graph classifier) G-Safeguard (Wang et al., 2025)
Graph detector run.guardian_lite GUARDIAN-Lite (attribute-reconstruction autoencoder) GUARDIAN (Zhou et al., 2025)
Graph detector run.pygod_anomalydae AnomalyDAE dual autoencoder (PyGOD) AnomalyDAE (Fan et al., 2020)
Graph detector run.pygod_conad CONAD contrastive graph autoencoder (PyGOD) CONAD (Xu et al., 2022)
Graph detector run.pygod_dominant DOMINANT graph autoencoder (PyGOD) DOMINANT (Ding et al., 2019)
Graph detector run.pygod_gaan GAAN adversarial detector (PyGOD) GAAN (Chen et al., 2020)
Baseline run.random Random run scores CatchBench (2026)

Step: Which Step Was the Decisive Mistake?

locator.rank(trace) orders the steps of a failed run, most suspected first, and locator.step_scores(trace) gives their scores. Supervised locators are first fitted with fit(traces, y), where y holds the idx of each run's mistake. The step-by-step and binary-search LLM judges name one step with predict_step(trace).

Type Method Algorithm Reference
Rule step.blast Blast radius (downstream reach) CatchBench (2026)
Rule step.degree Dependency degree CatchBench (2026)
Rule step.dep_anomaly Dependency anomaly (within-run z-score) CatchBench (2026)
Rule step.has_dep Has a dependency CatchBench (2026)
Rule step.max_span Maximum dependency span CatchBench (2026)
Rule step.position_early_rule Early-position prior CatchBench (2026)
Supervised step.exec_rank Execution-feature step ranker GRADE (Zhao, 2026)
Supervised step.position_supervised Early-fault position prior GRADE (Zhao, 2026)
Graph detector step.pygod_dominant_sg4 DOMINANT on the step dependency graph DOMINANT (Ding et al., 2019)
Graph detector step.pygod_dominant_ww8 DOMINANT on the Who&When step matrix DOMINANT (Ding et al., 2019)
LLM judge step.llm_all_at_once LLM judge, all-at-once Who&When (Zhang et al., 2025)
LLM judge step.llm_binary_search LLM judge, binary-search Who&When (Zhang et al., 2025)
LLM judge step.llm_step_by_step LLM judge, step-by-step Who&When (Zhang et al., 2025)
Baseline step.random Random step scores CatchBench (2026)

Install

pip install auditable                  # core: the registry and the configuration rules, no third-party dependencies
pip install "auditable[bench]"         # the leaderboards: PyOD, scikit-learn, NumPy, NetworkX, and agentpile with its Hugging Face loaders
pip install "auditable[tabular-ad]"    # run, step, and online methods on trace features (NumPy, scikit-learn, PyOD, NetworkX)
pip install "auditable[agentpile]"     # read public datasets, trace files and captured runs through agentpile

The examples, tests and benchmark harnesses are in the repository, not the wheel; to run them, install from a clone with pip install -e ".[dev,tabular-ad,bench]".

The LLM judges take any prompt -> reply function as their backend; OpenAICompatibleBackend reaches an OpenAI-compatible endpoint with the standard library alone.

The graph methods (PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite), and so the graph rows of the leaderboards, need torch, PyTorch Geometric, PyGOD, and a sampling backend built for the installed torch, which pip cannot resolve from the extra; they need Python 3.10 or newer, since torch 2.12.1 and pyg-lib publish no wheels for 3.9. Install torch and the matching pyg-lib wheel first (the same steps on Linux, macOS on Apple silicon, and Windows):

pip install "torch==2.12.1" --index-url https://download.pytorch.org/whl/cpu
pip install pyg_lib -f https://data.pyg.org/whl/torch-2.12.1+cpu.html
pip install "auditable[graph-ad]"
python -c "import torch_geometric.typing as t; assert t.WITH_PYG_LIB"

Where torch is already installed, as on Colab, keep it: skip the first line, and in the pyg-lib URL replace 2.12.1+cpu with the installed version, which python -c "import torch; print(torch.__version__)" prints; the pinned CPU build would replace Colab's torch and break its preinstalled torchvision. PyG does not publish wheels for every patch release, so when pip finds no pyg_lib at that URL, use the .0 release with the same CUDA tag: Colab's torch 2.14.1+cu130 takes https://data.pyg.org/whl/torch-2.14.0+cu130.html. The graph-ad install was checked for 0.4.0 on a Colab T4 runtime with its torch 2.14.1+cu130, and for 0.3.0 on Colab CPU and T4 runtimes with Colab's torch and on Linux aarch64 (DGX Spark) with the pinned one; a weekly workflow (bench-platforms.yml) repeats the pinned steps from a clean install on Linux, macOS, and Windows and checks three boards against their golden files. The PyGOD methods run on the CPU (gpu=-1) even when a GPU is present.

Traces and Inputs

Methods read traces through auditable.featurize.normalize_trace, which accepts a list of step dicts, a run dict ({"steps": ..., "texts": ...}), a typed NetworkX graph from auditable.graph.build_graph, a SessionGraph, or the TraceView that from_agentpile returns. Steps carry idx, agent, kind ("decision" or "tool_call"), and optionally deps and text; a SessionGraph also keeps parallel branches and merges of the execution order. A trace without dependency annotations can be read with dependency="full_context", the assumption that each step relied on every earlier step. A method that needs a field the trace lacks (step text for an LLM judge, for example) raises an error naming it.

auditable.featurize exposes the features the methods compute, so they can be inspected or reused: the GRADE structural layers (grade_layer, grade_layer_matrix), the Who&When step matrix (who_when_step_matrix), the step dependency graph and its statistics (step_graph, span_series, run_max_span), and graph encodings for PyGOD (to_pyg).

Reproducing CatchBench

python -m auditable.bench <task> --golden reruns a CatchBench board with the methods of this library and exits 1 if a row differs from the board (Leaderboard). The parity harness benchmarks/catchbench_parity remains for development and is not part of the wheel. It compares every migrated method with its CatchBench or GRADE original sample by sample, at CatchBench 0.1.2 and GRADE commit 3839a57, the last versions whose board computes those entries with its own code.

export GRADE_DIR=/path/to/grade   # checked out at 3839a57
export PYTHONHASHSEED=0
python benchmarks/catchbench_parity/run_parity.py checks                     # per-sample comparisons
python benchmarks/catchbench_parity/run_parity.py board --out board.txt      # the full board through the library
python benchmarks/catchbench_parity/run_parity.py board --perturb --out perturbed.txt
python benchmarks/catchbench_parity/run_parity.py consumption board.txt perturbed.txt

Adding a Method

Subclass the family base class in auditable.base, give it a MethodCard, and register it with @register, which makes it reachable by id (get_method, --methods) and adds it to the default rows of the detect and pre boards. To try a method first, pass it to leaderboard(task, extra=[...]) without registering (Put Your Method on the Board). The method contract lists the conventions: input forms, label alignment for supervised step methods, output order, optional dependencies, the tests a method needs, and how to add a parity check when the method reproduces a CatchBench row.

Records and Graph Analysis

The package also contains the modules it was first built around. audit and replay capture an agent decision with the dependency state it relied on and re-decide it against later state; analyze_run ranks the steps of a finished run by structural blast share; analyze_plan lints a declared plan; LiveSession scores a growing run; and auditable.integrations captures observed dependency edges from LangGraph, LangChain, MCP servers, or any loop through TouchRecorder. See examples/README.md and the documentation.

Records are content-addressed and hash-chained, and that property is limited. A signature or digest authenticates only the entries that were submitted, a hash chain exposes a later edit only when its head is anchored with an independent party, and neither reveals an event that was never written. The chain therefore does not establish that a log is complete.

Legacy

ActionGate, ReferenceLedger (the in-process reference rail), and DataAuditor are deprecated as of 0.3 and emit a DeprecationWarning when constructed. They remain importable and will be removed in a later release. Tabular anomaly detection on dependency snapshots belongs in PyOD.

Related Projects

  • CatchBench: the paper and benchmark whose board the leaderboards reproduce (arXiv:2608.22808).
  • agentpile: the converter that loads public agent-trace datasets, trace files and captured runs into one record, which from_agentpile reads.
  • GRADE: the typed two-layer graph of agent execution and dependency that the structural features read.
  • awesome-auditable-ai: a reading list on AI agent reliability and auditing.

Citation

If you use auditable, please cite Auditable Agents:

@misc{nian2026auditable,
  title         = {Auditable Agents},
  author        = {Nian, Yi and Yuan, Aojie and Zhang, Haiyue and Li, Jiate and Li, Li and Hu, Xiyang and Wei, Hua and Xiao, Xiongye and Xiao, Chaowei and Zhao, Yue},
  year          = {2026},
  eprint        = {2604.05485},
  archivePrefix = {arXiv}
}

If you use the structural features or the benchmark, please also cite GRADE and CatchBench, and for each method, the work behind it (its MethodCard.cite):

@article{zhao2026grade,
  title   = {GRADE: Graph Representation of LLM Agent Dependency and Execution},
  author  = {Zhao, Yue},
  journal = {arXiv preprint arXiv:2606.22741},
  year    = {2026}
}

@misc{zhao2026catchbench,
  title         = {{CatchBench}: When Can an Agent Failure Be Caught?},
  author        = {Yue Zhao},
  year          = {2026},
  eprint        = {2608.22808},
  archivePrefix = {arXiv}
}

License

Apache-2.0. Ported code keeps the notice of its source (GRADE and CatchBench, both MIT License) at the top of each module.