Skip to content
Projects

rzhub/GateMem has 136 GitHub stars and 4 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-16 18:54 UTC. Last push 2026-10-12.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

GateMem: a benchmark and evaluation toolkit for memory governance in multi-principal shared-memory LLM agents.

AI

README.md

Original

GateMem logo

Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents

GateMem evaluates whether memory-augmented LLM agents can remain useful, enforce access control, and honor deletion requests in multi-principal shared-memory environments.

📄 Paper • 🤗 Hugging Face Dataset • 🧪 Benchmark Toolkit • 🗂 Dataset Card • 🏆 Leaderboard • 🌐 Project Page

⭐ If you find GateMem useful, please consider starring the repository to help others discover the benchmark.


GateMem overview

GateMem evaluates memory governance in multi-principal shared-memory agents, including utility, access control, and active forgetting.


✨ What is GateMem?

Most memory benchmarks ask a simple question:

Can the agent remember correctly?

GateMem asks a harder and more deployment-relevant one:

Can the agent govern shared memory correctly across multiple principals, roles, scopes, and deletion requests?

This matters in settings like:

  • 🩺 Medical: patient / clinician / family coordination / ...
  • 💼 Office: manager / HR / employee / contractor workflows / ...
  • 🎓 Education: student / teacher / counselor / staff interactions / ...
  • 🏠 Household: family / guest / caregiver / resident coordination / ...

GateMem evaluates three tightly coupled capabilities:

Capability What it tests
Utility Can the agent answer correctly for authorized requests?
Access Control Can the agent avoid leaking protected information to unauthorized or over-scoped requesters?
Active Forgetting Can the agent avoid recovering or confirming deleted information after explicit deletion requests?

📌 Benchmark at a Glance

Property Value
Domains Medical, Office, Education, Household
Episodes 91 long-form multi-party episodes
Checkpoints 2,218 hidden checkpoints
Evaluation categories Utility / Access Control / Active Forgetting
Baselines included Long-Context, RAG-Naive, RAG-Policy, A-MEM, Mem0, ReMeM-I, ReMeM-S
Main summary metric Memory Governance Score (MGS)

Main metric:

MGS = U * (1 - A) * (1 - F)

where:

  • U = Utility
  • A = Access-Control Violation Rate
  • F = Active-Forgetting Failure Rate

🧠 Why GateMem?

GateMem is designed for shared-memory agents, where memory is not a private cache but a common pool used by multiple principals.

A good shared-memory agent must do all three:

  • Be helpful when the request is legitimate
  • Be safe when the request exceeds authorization
  • Be forgetful when information has been explicitly deleted

This makes GateMem different from prior benchmarks focused mainly on recall, personalization, or long-horizon memory utility.


🏗️ Benchmark Construction Pipeline

GateMem pipeline

GateMem is built from domain-specific scenario specifications, long-form multi-party episodes, and hidden checkpoints for utility, access control, and active forgetting.


📊 Main Takeaway

Across diverse backbones and memory-agent baselines, no method simultaneously achieves strong utility, robust access control, and reliable active forgetting.

  • Long-context prompting often gives the strongest overall trade-off, but at substantial token cost.
  • Retrieval-based and external-memory methods reduce cost, but still exhibit non-trivial leakage and post-deletion recovery failures.
  • Shared-memory governance remains a challenging open problem for memory-augmented LLM agents.

GateMem main results

Across diverse backbones and memory-agent baselines, no method simultaneously achieves strong utility, robust access control, and reliable active forgetting.


🚀 Quick Start

Install dependencies:

pip install -r requirements.txt

Set API keys if you use external LLMs:

export OPENAI_API_KEY="..."
export DEEPSEEK_API_KEY="..."
export GEMINI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
export NVIDIA_API_KEY="..."

Run a simple baseline on the medical domain:

python bench/scripts/run_eval.py \
  --data_dir bench/data/medical \
  --agent long_context

Run Policy RAG with an answer model and LLM judge:

python bench/scripts/run_eval.py \
  --data_dir bench/data/medical \
  --agent rag_policy \
  --llm_provider openai \
  --llm_model gpt-4o-mini \
  --temperature 0.2 \
  --max_output_tokens 4096 \
  --use_llm_judge \
  --judge_provider openai \
  --judge_model gpt-4o \
  --judge_concurrency 4

For full benchmark usage, see bench/README.md.


🔬 Evaluate Your Own Method

GateMem provides an online submission interface for validating pre-scored result packages and updating the public leaderboard.

Submit results: 👉 GateMem-Submit
View leaderboard: 🏆 GateMem Leaderboard

Workflow

  1. Generate a complete predictions.jsonl file for one GateMem domain.
  2. Run the official evaluator locally with the official LLM judge configuration and your own API credentials.
  3. Package predictions.jsonl, predictions.normalized.jsonl, scores.jsonl, judge_scores.jsonl, and summary.json as submission.zip.
  4. Upload submission.zip to the GateMem Submission Space and provide the method metadata.
  5. After maintainer approval, the result is published as Community. A result becomes Verified only after the maintainers re-score the exact same predictions.

The public Space validates and records submitted artifacts; evaluation is performed locally. See docs/prediction_format.md for the scoring and packaging commands.

Verification Status

  • Community: evaluated locally with the official scripts and submitted by the contributor; not yet independently re-scored by the maintainers.
  • Verified: the maintainers have re-scored the exact submitted predictions.jsonl and published the resulting official metrics.

Prediction Format

The evaluator input is a JSONL file named predictions.jsonl, with one prediction per checkpoint. The online interface accepts the complete submission.zip produced after local scoring.

See docs/prediction_format.md.

Implementing a New Agent

To implement a method inside this repository, see:

Once predictions.jsonl is generated, score and package it as described in docs/prediction_format.md, then upload submission.zip through the online interface above.


📁 Repository Structure

.
├── README.md
├── DATASET_CARD.md
├── CITATION.cff
├── bench/
│   ├── README.md
│   ├── data/
│   │   ├── medical/
│   │   ├── office/
│   │   ├── education/
│   │   └── household/
│   ├── agents/
│   ├── scripts/
│   └── ...
├── docs/
│   ├── adding_new_agent.md
│   ├── prediction_format.md
│   ├── evaluation_protocol.md
│   └── reproduce_paper.md
├── configs/
└── scripts/

📝 Notes on Evaluation

The paper uses the terms:

  • Utility
  • Access Control
  • Active Forgetting

For backward compatibility, the code and released JSON files keep legacy names:

Paper term Code / data name
Utility utility
Access Control privacy
Active Forgetting safety

Corresponding metrics:

  • utility_accuracy → Utility U
  • privacy_leakage_rate → Access-Control Violation A
  • deletion_leakage_rate → Active-Forgetting Failure F
  • compliance_utility_score → Memory Governance Score MGS

⚠️ Release Note

GateMem is released as an open-label offline benchmark for transparent research and method development.

During evaluation, methods should not use hidden annotation fields such as:

  • query_type
  • expected_action
  • judge_spec
  • leak_targets

These fields are intended for scoring and analysis only.


📚 Citation

If you use GateMem, please cite the accompanying paper.

@misc{ren2026gatemembenchmarkingmemorygovernance,
      title={GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents}, 
      author={Zhe Ren and Yibo Yang and Yimeng Chen and Zijun Zhao and Benshuo Fu and Zhihao Shu and Bingjie Zhang and Yangyang Xu and Dandan Guo and Shuicheng Yan},
      year={2026},
      eprint={2606.18829},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2606.18829}, 
}