Explainable Reinforcement Learning (XRL) Resources

Explainable reinforcement learning (XRL) aims to make the goals, decisions, and learned behavior of RL agents understandable to humans. This repository is a curated, comprehensive bibliography of the field: 18 surveys, 261 articles, 10 PhD theses, and 1 overview published between 2016 and 2026.

Topics include interpretable policies, saliency and attribution methods, counterfactual explanations, policy and trajectory summarization, reward decomposition, and programmatic or neuro-symbolic agents.

⭐ Star the repo to help others find it. Open an issue to suggest a paper or report an error, or reach out at contact (at) xrl (dot) ai.

How to Use This List

New to XRL? Begin with the General XRL Surveys under Survey Papers for a grounded overview of the field, and see PhD Theses for in-depth treatments of individual lines of work. To keep up with current work, skim Recently Added. To find work on a specific technique, use the Table of Contents to jump to its topic under Papers, where each table lists the newest papers first.

Table of Contents

• Resources
• Survey Papers (18 papers)
  • General XRL Surveys (9)
  • Focused Surveys (4)
  • Application Domains (3)
  • Frameworks and Perspectives (2)
• PhD Theses (10 theses)
• Papers (261 papers)
  • Decision Trees and Rules (39)
  • Programmatic and Symbolic Policies (30)
  • Concepts and Learned Representations (39)
  • Saliency and Feature Attribution (48)
  • Counterfactual and Contrastive Explanations (17)
  • Policy Summarization and Critical States (34)
  • Reward and Value Explanations (17)
  • Causal and Outcome Explanations (11)
  • Natural Language Explanations (10)
  • Visual Analytics (12)
  • Human Studies and Evaluation (4)

Recently Added

• A survey of explainable reinforcement learning: targets, methods and needs — J. Artif. Intell. Res., 2026
• Selecting Decision-Relevant Concepts in Reinforcement Learning — CoRR, 2026
• Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent — Mechanistic Interpretability Workshop @ ICML, 2024
• Making Teams and Influencing Agents: Efficiently Coordinating Decision Trees for Interpretable Multi-Agent Reinforcement Learning — AIES, 2025
• MAVIPER: Learning Decision Tree Policies for Interpretable Multi-agent Reinforcement Learning — ECML PKDD, 2023
• Aligning Agent Policies with Preferences: Human-Centered Interpretable Reinforcement Learning — AIES, 2025
• LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning — ICLR, 2025
• Interpretable Deep Reinforcement Learning Via Concept-Based Policy Distillation — Mach. Learn., 2025
• Hierarchical Programmatic Option Framework — NeurIPS, 2024
• Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents — NeurIPS, 2024

Resources

• Awesome Explainable Reinforcement Learning — curated list of XRL papers and resources
• AAAI 2024 Workshop on eXplainable AI approaches for Deep Reinforcement Learning (XAI4DRL) — workshop proceedings on OpenReview
• RLC 2024 Workshop on Interpretable Policies in Reinforcement Learning (InterpPol) — workshop proceedings on OpenReview
• RLC 2025 Workshop on Programmatic Reinforcement Learning (PRL) — workshop proceedings on OpenReview
• Explanations for Sequential Decision-Making - an Overview — AAAI 2026 overview by Baier et al.

Survey Papers

General XRL Surveys

#/Link	Title	Venue/Journal	Year
1	A survey of explainable reinforcement learning: targets, methods and needs	J. Artif. Intell. Res.	2026
2	Explainable Reinforcement Learning: A Survey and Comparative Review	ACM Comput. Surv.	2024
3	Explainable reinforcement learning (XRL): a systematic literature review and taxonomy	Mach. Learn.	2024
4	A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges	CoRR	2023
5	Explainable Deep Reinforcement Learning: State of the Art and Challenges	ACM Comput. Surv.	2022
6	Explainable AI and Reinforcement Learning - A Systematic Review of Current Approaches and Trends	Frontiers Artif. Intell.	2021
7	Explainability in deep reinforcement learning	Knowl. Based Syst.	2021
8	Explainable Reinforcement Learning: A Survey	CD-MAKE	2020
9	Reinforcement Learning Interpretation Methods: A Survey	IEEE Access	2020

Back to top

Focused Surveys

#/Link	Title	Venue/Journal	Year
1	A Survey on Explainable Deep Reinforcement Learning	CoRR	2025
2	Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities	ACM Comput. Surv.	2024
3	A Survey of Global Explanations in Reinforcement Learning	Explainable Agency in Artificial Intelligence	2024
4	A survey on interpretable reinforcement learning	Mach. Learn.	2024

Back to top

Application Domains

#/Link	Title	Venue/Journal	Year
1	Explainable and Interpretable Reinforcement Learning for Robotics	SLAIML	2024
2	Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications	ACM Comput. Surv.	2023
3	Advances in Explainable Reinforcement Learning: An Intelligent Transportation Systems Perspective	Explainable Artificial Intelligence for Intelligent Transportation Systems	2023

Back to top

Frameworks and Perspectives

#/Link	Title	Venue/Journal	Year
1	Explainable reinforcement learning for broad-XAI: a conceptual framework and survey	Neural Comput. Appl.	2023
2	Explainability in reinforcement learning: perspective and position	CoRR	2022

Back to top

PhD Theses

#/Link	Title	Author	University	Year
1	Human-Centered Reinforcement Learning	Stephanie Milani	Carnegie Mellon University	2025
2	Explainable Reinforcement Learning (XRL): Simplifying Agent Behavior	Yanzhe Bekkemoen	Norwegian University of Science and Technology, Trondheim, Norway	2025
3	Counterfactual and Semifactual Explanations for Reinforcement Learning	Jasmina Gajcin	Trinity College Dublin, School of Computer Science & Statistics	2025
4	Explaining the future context of deep reinforcement learning agents' decision-making	Mark Towers	University of Southampton	2025
5	On the Interpretability of Reinforcement Learning	Herman Yau	University of Surrey	2025
6	Interprétabilité, arbres de décision, prise de décision séquentielle	Hector Kohler	Université de Lille	2025
7	Tree Models for Interpretable Agents	Tom Bewley	University of Bristol	2024
8	Towards a combined local and global explanation framework for deep reinforcement learning agents with visual input: novel methods and insights from human evaluation	Tobias Huber	Universität Augsburg	2024
9	Explainable Reinforcement Learning Through a Causal Lens	Prashan Madumal	The University of Melbourne	2021
10	Interpretable Reinforcement Learning Policies by Evolutionary Computation	Daniel Hein	Technische Universität München	2019

Back to top

Papers

Top venues: NeurIPS (19) | ICLR (16) | AAAI (11) | ICML (10) | AAMAS (9) | CoRR (9) | XAI Workshop @ IJCAI (5) | AIES (4) | Artif. Intell. (4) | ECML PKDD (4) | IEEE Robotics Autom. Lett. (4) | IJCAI (4) | IROS (4) | SSCI (4) | Eng. Appl. Artif. Intell. (3)

Decision Trees and Rules

#/Link	Title	Venue/Journal	Year
1	Aligning Agent Policies with Preferences: Human-Centered Interpretable Reinforcement Learning	AIES	2025
2	Making Teams and Influencing Agents: Efficiently Coordinating Decision Trees for Interpretable Multi-Agent Reinforcement Learning	AIES	2025
3	Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning	CoRR	2024
4	MAVIPER: Learning Decision Tree Policies for Interpretable Multi-agent Reinforcement Learning	ECML PKDD	2023
5	Inherently Interpretable Deep Reinforcement Learning Through Online Mimicking	EXTRAAMAS	2023
6	Extracting Decision Tree From Trained Deep Reinforcement Learning in Traffic Signal Control	IEEE Trans. Comput. Soc. Syst.	2023
7	Model tree methods for explaining deep reinforcement learning agents in real-time robotic applications	Neurocomputing	2023
8	Enhanced Oblique Decision Tree Enabled Policy Extraction for Deep Reinforcement Learning in Power System Emergency Control	Electric Power Systems Research	2022
9	POETREE: Interpretable Policy Learning with Adaptive Decision Trees	ICLR	2022
10	Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems	IEEE Transactions on Cybernetics	2022
11	Mo"ET: Mixture of Expert Trees and its application to verifiable reinforcement learning	Neural Networks	2022
12	Learning Interpretable, High-Performing Policies for Autonomous Driving	Robotics: Science and Systems	2022
13	Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods	AAAI	2021
14	TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environments	AAAI	2021
15	Explaining Deep Reinforcement Learning Agents in the Atari Domain through a Surrogate Model	AIIDE	2021
16	CDT: Cascading Decision Trees for Explainable Reinforcement Learning	CoRR	2021
17	Approximating a deep reinforcement learning docking agent using linear model trees	ECC	2021
18	Explainable Reinforcement Learning with the Tsetlin Machine	IEA/AIE	2021
19	Explaining Deep Learning Models Through Rule-Based Approximation and Visualization	IEEE Trans. Fuzzy Syst.	2021
20	XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees	IROS	2021
21	Mixed Autonomous Supervision in Traffic Signal Control	ITSC	2021
22	Can You Trust Your Autonomous Car? Interpretable and Verifiably Safe Reinforcement Learning	IV	2021
23	Explaining a Deep Reinforcement Learning Docking Agent Using Linear Model Trees with User Adapted Visualization	Journal of Marine Science and Engineering	2021
24	Learning Tree Interpretation from Object Representation for Deep Reinforcement Learning	NeurIPS	2021
25	A co-evolutionary approach to interpretable reinforcement learning in environments with continuous action spaces	SSCI	2021
26	Interpretable AI Agent Through Nonlinear Decision Trees for Lane Change Problem	SSCI	2021
27	Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning	AISTATS	2020
28	Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data	Challenges of Real-World RL Workshop @ NeurIPS	2020
29	Interpretable policies for reinforcement learning by empirical fuzzy sets	Eng. Appl. Artif. Intell.	2020
30	Improved Policy Extraction via Online Q-Value Distillation	IJCNN	2020
31	Explainable navigation system using fuzzy reinforcement learning	IJIDeM	2020
32	Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving	Robotics Auton. Syst.	2020
33	Modelling Agent Policies with Interpretable Imitation Learning	TAILOR	2020
34	Interpretable Approximation of a Deep Reinforcement Learning Agent as a Set of If-Then Rules	ICMLA	2019
35	Distilling Deep Reinforcement Learning Policies in Soft Decision Trees	XAI Workshop @ IJCAI	2019
36	Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees	ECML PKDD	2018
37	Generating interpretable fuzzy controllers using particle swarm optimization and genetic programming	GECCO	2018
38	Verifiable Reinforcement Learning via Policy Extraction	NeurIPS	2018
39	Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies	Eng. Appl. Artif. Intell.	2017

Back to top

Programmatic and Symbolic Policies

#/Link	Title	Venue/Journal	Year
1	Neural DNF-MT: A Neuro-symbolic Approach for Learning Interpretable and Editable Policies	AAMAS	2025
2	From Explainability to Interpretability: Interpretable Reinforcement Learning Via Model Explanations	RLC	2025
3	End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations	ICML	2024
4	Hierarchical Programmatic Option Framework	NeurIPS	2024
5	Learning of Generalizable and Interpretable Knowledge in Grid-Based Reinforcement Learning Environments	AIIDE	2023
6	Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach	ECML PKDD	2023
7	Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction	NeurIPS	2023
8	Multi-objective Genetic Programming for Explainable Reinforcement Learning	EuroGP	2022
9	Deep-Learning-based Fuzzy Symbolic Processing with Agents Capable of Knowledge Communication	ICAART	2022
10	Programmatic Reinforcement Learning without Oracles	ICLR	2022
11	Hierarchical Program-Triggered Reinforcement Learning Agents for Automated Driving	IEEE Trans. Intell. Transp. Syst.	2022
12	GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis	NeurIPS	2022
13	DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learning	AAAI	2021
14	Off-Policy Differentiable Logic Reinforcement Learning	ECML PKDD	2021
15	Neuro-Symbolic Reinforcement Learning with First-Order Logic	EMNLP	2021
16	Discovering symbolic policies with deep reinforcement learning	ICML	2021
17	Symbolic Regression Methods for Reinforcement Learning	IEEE Access	2021
18	Programmatic Policy Extraction by Iterative Local Search	ILP	2021
19	Automatic discovery of interpretable planning strategies	Mach. Learn.	2021
20	Learning to Synthesize Programs as Interpretable and Generalizable Policies	NeurIPS	2021
21	Learning an Interpretable Traffic Signal Control Policy	AAMAS	2020
22	Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis	Complex & Intelligent Systems	2020
23	Explainability of Intelligent Transportation Systems using Knowledge Compilation: a Traffic Light Controller Case	ITSC	2020
24	Interpretable, Verifiable, and Robust Reinforcement Learning via Program Synthesis	xxAI - Beyond Explainable AI	2020
25	SDRL: Interpretable and Data-Efficient Deep Reinforcement Learning Leveraging Symbolic Planning	AAAI	2019
26	Neural Logic Reinforcement Learning	ICML	2019
27	Imitation-Projected Programmatic Reinforcement Learning	NeurIPS	2019
28	A formal methods approach to interpretable reinforcement learning for robotic planning	Sci. Robotics	2019
29	Interpretable policies for reinforcement learning by genetic programming	Eng. Appl. Artif. Intell.	2018
30	Programmatically Interpretable Reinforcement Learning	ICML	2018

Back to top

Concepts and Learned Representations

#/Link	Title	Venue/Journal	Year
1	Selecting Decision-Relevant Concepts in Reinforcement Learning	CoRR	2026
2	Interpreting Emergent Planning in Model-Free Reinforcement Learning	ICLR	2025
3	LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning	ICLR	2025
4	Inducing, Detecting and Characterising Neural Modules: A Pipeline for Functional Interpretability in Reinforcement Learning	ICML	2025
5	Interpretable Deep Reinforcement Learning Via Concept-Based Policy Distillation	Mach. Learn.	2025
6	Discovering Behavioral Modes in Deep Reinforcement Learning Policies Using Trajectory Clustering in Latent Space	CoRR	2024
7	Deep Reinforcement Learning Behavioral Mode Switching Using Optimal Control Based on a Latent Space Objective	MED	2024
8	Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent	Mechanistic Interpretability Workshop @ ICML	2024
9	Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents	NeurIPS	2024
10	Information based explanation methods for deep learning agents--with applications on large open-source chess models	Scientific Reports	2024
11	Interpreting a deep reinforcement learning model with conceptual embedding and performance analysis	Appl. Intell.	2023
12	Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero	CoRR	2023
13	Towards Interpretable Deep Reinforcement Learning with Human-Friendly Prototypes	ICLR	2023
14	Unveiling Concepts Learned by a World-Class Chess-Playing Agent	IJCAI	2023
15	Extracting tactics learned from self-play in general games	Inf. Sci.	2023
16	State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding	NeurIPS	2023
17	Concept Learning for Interpretable Multi-Agent Reinforcement Learning	CoRL	2022
18	InAction: Interpretable Action Decision Making for Autonomous Driving	ECCV	2022
19	Interpretable Autonomous Flight Via Compact Visualizable Neural Circuit Policies	IEEE Robotics Autom. Lett.	2022
20	Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning	IEEE Trans. Intell. Transp. Syst.	2022
21	Continuous Action Reinforcement Learning From a Mixture of Interpretable Experts	IEEE Trans. Pattern Anal. Mach. Intell.	2022
22	ProtoX: Explaining a Reinforcement Learning Agent via Prototyping	NeurIPS	2022
23	Analysing deep reinforcement learning agents trained with domain randomisation	Neurocomputing	2022
24	Acquisition of chess knowledge in AlphaZero	Proc. Natl. Acad. Sci. U.S.A.	2022
25	Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning	ICLR	2021
26	Re-understanding Finite-State Representations of Recurrent Policy Networks	ICML	2021
27	Efficient Robotic Object Search Via HIEM: Hierarchical Policy Learning With Intrinsic-Extrinsic Modeling	IEEE Robotics Autom. Lett.	2021
28	Interpretable Decision-Making for Autonomous Vehicles at Highway On-Ramps With Latent Space Reinforcement Learning	IEEE Trans. Veh. Technol.	2021
29	Model primitives for hierarchical lifelong reinforcement learning	Auton. Agents Multi Agent Syst.	2020
30	Understanding RL Vision	Distill	2020
31	Towards Interpretable Reinforcement Learning with State Abstraction Driven by External Knowledge	IEICE Trans. Inf. Syst.	2020
32	Visualization of topographical internal representation of learning robots	IJCNN	2020
33	Towards Governing Agent's Efficacy: Action-Conditional \textdollar(\beta)\textdollar-VAE for Deep Transparent Reinforcement Learning	ACML	2019
34	Deep reinforcement learning with relational inductive biases	ICLR	2019
35	Learning Finite State Representations of Recurrent Policy Networks	ICLR	2019
36	Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation	IROS	2019
37	Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning	ICLR	2018
38	Unsupervised Video Object Segmentation for Deep Reinforcement Learning	NeurIPS	2018
39	Graying the black box: Understanding DQNs	ICML	2016

Back to top

Saliency and Feature Attribution

#/Link	Title	Venue/Journal	Year
1	Demystifying Reinforcement Learning in Production Scheduling via Explainable AI	CoRR	2024
2	Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning	IEEE Trans. Syst. Man Cybern. Syst.	2024
3	Explaining Deep Reinforcement Learning Policies with SHAP, Decision Trees, and Prototypes	MED	2024
4	Explaining Reinforcement Learning with Shapley Values	ICML	2023
5	Achieving efficient interpretability of reinforcement learning via policy distillation and selective input gradient regularization	Neural Networks	2023
6	Explaining Deep Reinforcement Learning-Based Methods for Control of Building HVAC Systems	xAI	2023
7	Energy-Efficient Driving for Adaptive Traffic Signal Control Environment via Explainable Reinforcement Learning	Applied Sciences	2022
8	Attributation Analysis of Reinforcement Learning-Based Highway Driver	Electronics	2022
9	Mean-variance Based Risk-sensitive Reinforcement Learning with Interpretable Attention	ICMVA	2022
10	Explainable AI in Deep Reinforcement Learning Models for Power System Emergency Control	IEEE Trans. Comput. Soc. Syst.	2022
11	Self-Supervised Discovering of Interpretable Features for Reinforcement Learning	IEEE Trans. Pattern Anal. Mach. Intell.	2022
12	Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning	IEEE Trans. Pattern Anal. Mach. Intell.	2022
13	Understanding via Exploration: Discovery of Interpretable Features With Deep Reinforcement Learning	IEEE Transactions on Neural Networks and Learning Systems	2022
14	Incorporating Explanations to Balance the Exploration and Exploitation of Deep Reinforcement Learning	KSEM	2022
15	Towards Explainable Reinforcement Learning Using Scoring Mechanism Augmented Agents	KSEM	2022
16	Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture	Top. Cogn. Sci.	2022
17	Explainable Deep Reinforcement Learning for UAV autonomous path planning	Aerospace Science and Technology	2021
18	XPM: An Explainable Deep Reinforcement Learning Framework for Portfolio Management	CIKM	2021
19	Robotic Lever Manipulation using Hindsight Experience Replay and Shapley Additive Explanations	ECC	2021
20	Explainable Reinforcement Learning for Longitudinal Control	ICAART	2021
21	Explainable deep reinforcement learning for portfolio management: an empirical approach	ICAIF	2021
22	DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation	ICCV	2021
23	A Blood Glucose Control Framework Based on Reinforcement Learning With Safety and Interpretability: In Silico Validation	IEEE Access	2021
24	Learning to Discover Task-Relevant Features for Interpretable Reinforcement Learning	IEEE Robotics Autom. Lett.	2021
25	Explainable AI methods on a deep reinforcement learning agent for automatic docking	IFAC-PapersOnLine	2021
26	Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning	IJCNN	2021
27	Explaining the Decisions of Deep Policy Networks for Robotic Manipulations	IROS	2021
28	Machine versus Human Attention in Deep Reinforcement Learning Tasks	NeurIPS	2021
29	Feature-Based Interpretable Reinforcement Learning based on State-Transition Models	SMC	2021
30	Attribution-based Salience Method towards Interpretable Reinforcement Learning	AAAI-MAKE	2020
31	Methodology for Interpretable Reinforcement Learning Model for HVAC Energy Control	Big Data	2020
32	Explaining Autonomous Driving by Learning End-to-End Visual Attention	CVPRW	2020
33	Neuroevolution of self-interpretable agents	GECCO	2020
34	Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution	ICLR	2020
35	Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning	ICLR	2020
36	Deep Reinforcement Learning for Safe Local Planning of a Ground Vehicle in Unknown Rough Terrain	IEEE Robotics Autom. Lett.	2020
37	xGAIL: Explainable Generative Adversarial Imitation Learning for Explainable Human Decision Analysis	KDD	2020
38	Towards Better Interpretability in Deep Q-Networks	AAAI	2019
39	Visualization of Deep Reinforcement Learning using Grad-CAM: How AI Plays Atari Games?	CoG	2019
40	Free-Lunch Saliency via Attention in Atari Agents	ICCVW	2019
41	Visualizing Deep Q-Learning to Understanding Behavior of Swarm Robotic System	IES	2019
42	Reinforcement Learning with Explainability for Traffic Signal Control	ITSC	2019
43	Enhancing Explainability of Deep Reinforcement Learning Through Selective Layer-Wise Relevance Propagation	KI	2019
44	Towards Interpretable Reinforcement Learning Using Attention Augmented Agents	NeurIPS	2019
45	Transparency and Explanation in Deep Reinforcement Learning Neural Networks	AIES	2018
46	Visual Rationalizations in Deep Reinforcement Learning for Atari Games	BNAIC	2018
47	Visualizing and Understanding Atari Agents	ICML	2018
48	Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention	ICCV	2017

Back to top

Counterfactual and Contrastive Explanations

#/Link	Title	Venue/Journal	Year
1	Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes	AAAI	2024
2	Semifactual Explanations for Reinforcement Learning	Proc. of HAI	2024
3	GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations	AAMAS	2023
4	Counterfactual Explanation Policies in RL	Counterfactuals in Minds and Machines Workshop @ ICML	2023
5	Explaining Black Box Reinforcement Learning Agents Through Counterfactual Policies	IDA	2023
6	Real-Time Counterfactual Explanations For Robotic Systems With Multiple Continuous Outputs	IFAC-PapersOnLine	2023
7	Explainable Multi-Agent Reinforcement Learning for Temporal Queries	IJCAI	2023
8	Contrastive Visual Explanations for Reinforcement Learning via Counterfactual Rewards	xAI	2023
9	Explaining Reinforcement Learning Policies through Counterfactual Trajectories	HILL Workshop @ ICML	2022
10	Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations	ICLR	2022
11	Explainable Reinforcement Learning via Model Transforms	NeurIPS	2022
12	Counterfactual state explanations for reinforcement learning agents via generative deep learning	Artif. Intell.	2021
13	Contrastive Explanations for Comparing Preferences of Reinforcement Learning Agents	CoRR	2021
14	Contrastive Explanations for Reinforcement Learning via Embedded Self Predictions	ICLR	2021
15	Learning "What-if" Explanations for Sequential Decision-Making	ICLR	2021
16	Counterfactual States for Atari Agents via Generative Deep Learning	XAI Workshop @ IJCAI	2019
17	Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences	XAI Workshop @ IJCAI	2018

Back to top

Policy Summarization and Critical States

#/Link	Title	Venue/Journal	Year
1	ASAP: Attention-Based State Space Abstraction for Policy Summarization	ACML	2024
2	ASQ-IT: Interactive explanations for reinforcement-learning agents	Artif. Intell.	2024
3	Detection of Important States through an Iterative Q-value Algorithm for Explainable Reinforcement Learning	HICSS	2024
4	''You just can't go around killing people'' Explaining Agent Behavior to a Human Terminator	MHFAIA Workshop @ ICML	2024
5	Local Explanations for Reinforcement Learning	AAAI	2023
6	Learning state importance for preference-based reinforcement learning	Mach. Learn.	2023
7	StateMask: Explaining Deep Reinforcement Learning through State Mask	NeurIPS	2023
8	Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents	PAAMS	2023
9	IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit Based on Analyses of Interestingness	xAI	2023
10	"I Don't Think So": Summarizing Policy Disagreements for Agent Comparison	AAAI	2022
11	CAPS: Comprehensible Abstract Policy Summaries for Explaining Reinforcement Learning Agents	AAMAS	2022
12	Lazy-MDPs: Towards Interpretable RL by Learning When to Act	AAMAS	2022
13	Summarising and Comparing Agent Dynamics with Contrastive Spatiotemporal Abstraction	XAI Workshop @ IJCAI	2022
14	Explaining robot policies	Applied AI Letters	2021
15	Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps	Artif. Intell.	2021
16	Identifying Decision Points for Safe and Interpretable Reinforcement Learning in Hypotension Treatment	ML for Health Workshop @ NeurIPS	2021
17	EDGE: Explaining Deep Reinforcement Learning Policies	NeurIPS	2021
18	Learning Sparse Evidence- Driven Interpretation to Understand Deep Reinforcement Learning Agents	SSCI	2021
19	Interestingness elements for explainable reinforcement learning: Understanding agents' capabilities and limitations	Artif. Intell.	2020
20	Understanding the Behavior of Reinforcement Learning Agents	BIOMA	2020
21	Identifying Critical States by the Action-Based Variance of Expected Return	ICANN	2020
22	TLdR: Policy Summarization for Factored SSP Problems Using Temporal Abstractions	ICAPS	2020
23	Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents	ICLR	2020
24	Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions	ICML	2020
25	Generation of Policy-Level Explanations for Reinforcement Learning	AAAI	2019
26	Toward Robust Policy Summarization	AAMAS	2019
27	Summarizing agent strategies	Auton. Agents Multi Agent Syst.	2019
28	Enabling robots to communicate their objectives	Auton. Robots	2019
29	Exploring Computational User Models for Agent Policy Summarization	IJCA	2019
30	Interestingness Elements for Explainable Reinforcement Learning through Introspection	IUI Workshop	2019
31	HIGHLIGHTS: Summarizing Agent Behavior to People	AAMAS	2018
32	Deep Reinforcement Learning Monitor for Snapshot Recording	ICMLA	2018
33	Establishing Appropriate Trust via Critical States	IROS	2018
34	Visual Sparse Bayesian Reinforcement Learning: A Framework for Interpreting What an Agent Has Learned	SSCI	2018

Back to top

Reward and Value Explanations

#/Link	Title	Venue/Journal	Year
1	UTILITY: Utilizing Explainable Reinforcement Learning to Improve Reinforcement Learning	ICLR	2025
2	Causal State Distillation for Explainable Reinforcement Learning	CLeaR	2024
3	Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators	ECAI	2024
4	Explainable Reinforcement Learning Based on Q-Value Decomposition by Expected State Transitions	AAAI-MAKE	2023
5	A Closer Look at Reward Decomposition for High-Level Robotic Explanations	ICDL	2023
6	Hierarchical goals contextualize local reward decomposition explanations	Neural Comput. Appl.	2023
7	Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions	AAMAS	2022
8	Explaining Online Reinforcement Learning Decisions of Self-Adaptive Systems	ACSOS	2022
9	BEERL: Both Ends Explanations for Reinforcement Learning	Applied Sciences	2022
10	Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning	ICPR	2022
11	Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning	NeurIPS	2022
12	Explainable Reinforcement Learning for Human-Robot Collaboration	ICAR	2021
13	Understanding Learned Reward Functions	CoRR	2020
14	Explaining Reward Functions in Markov Decision Processes	FLAIRS	2019
15	Explanation-Based Reward Coaching to Improve Human Performance via Reinforcement Learning	HRI	2019
16	Explainable Reinforcement Learning via Reward Decomposition	XAI Workshop @ IJCAI	2019
17	Explaining Deep Adaptive Programs via Reward Decomposition	XAI Workshop @ IJCAI/ECAI	2018

Back to top

Causal and Outcome Explanations

#/Link	Title	Venue/Journal	Year
1	CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systems	AAAI	2024
2	Causal Explanations for Sequential Decision-Making in Multi-Agent Systems	AAMAS	2024
3	Explainable Reinforcement Learning via a Causal World Model	IJCAI	2023
4	Explainable robotic systems: understanding goal-driven actions in a reinforcement learning scenario	Neural Comput. Appl.	2023
5	Analysis of Explainable Goal-Driven Reinforcement Learning in a Continuous Simulated Environment	Algorithms	2022
6	A framework of explanation generation toward reliable autonomous robots	Adv. Robotics	2021
7	Explainable Artificial Intelligence (XAI) for Increasing User Trust in Deep Reinforcement Learning Driven Autonomous Systems	Deep RL Workshop @ NeurIPS	2021
8	Explainable Reinforcement Learning through a Causal Lens	AAAI	2020
9	What Did You Think Would Happen? Explaining Agent Behaviour through Intended Outcomes	NeurIPS	2020
10	Memory-Based Explainable Reinforcement Learning	AI	2019
11	Semantic Predictive Control for Explainable and Efficient Policy Learning	ICRA	2019

Back to top

Natural Language Explanations

#/Link	Title	Venue/Journal	Year
1	TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models	CoRR	2025
2	Explaining Intelligent Agent's Future Motion on Basis of Vocabulary Learning With Human Goal Inference	IEEE Access	2022
3	Inherently Explainable Reinforcement Learning in Natural Language	NeurIPS	2022
4	Driving behavior explanation with multi-level fusion	Pattern Recognit.	2022
5	Verbal Explanations for Deep Reinforcement Learning Neural Networks with Attention on Extracted Features	RO-MAN	2019
6	Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations	AIES	2018
7	Textual Explanations for Self-Driving Vehicles	ECCV	2018
8	Autonomous Self-Explanation of Behavior for Interactive Reinforcement Learning Agents	HAI	2017
9	Improving Robot Controller Transparency Through Autonomous Policy Explanation	HRI	2017
10	Application of Instruction-Based Behavior Explanation to a Reinforcement Learning Agent with Changing Policy	ICONIP	2017

Back to top

Visual Analytics

#/Link	Title	Venue/Journal	Year
1	Abstracted Trajectory Visualization for Explainability in Reinforcement Learning	CAI	2024
2	Comparing Strategies for Visualizing the High-Dimensional Exploration Behavior of CPS Design Agents	DESTION	2022
3	Visual Analytics for RNN-Based Deep Reinforcement Learning	IEEE Trans. Vis. Comput. Graph.	2022
4	ACMViz: a visual analytics approach to understand DRL-based autonomous control model	J. Vis.	2022
5	Why? Why not? When? Visual Explanations of Agent Behaviour in Reinforcement Learning	PacificVis	2022
6	Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning	Softw. Syst. Model.	2022
7	Interactive Explanations: Diagnosis and Repair of Reinforcement Learning Based Agent Behaviors	CoG	2021
8	Visual Analysis of Deep Q-network	KSII Trans. Internet Inf. Syst.	2021
9	DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning	Comput. Graph. Forum	2020
10	Topological Visualization Method for Understanding the Landscape of Value Functions and Structure of the State Space in Reinforcement Learning	ICAART	2020
11	DynamicsExplorer: Visual Analytics for Robot Control Tasks involving Dynamics and LSTM-based Control Policies	PacificVis	2020
12	DQNViz: A Visual Analytics Approach to Understand Deep Q-Networks	IEEE Trans. Vis. Comput. Graph.	2019

Back to top

Human Studies and Evaluation

#/Link	Title	Venue/Journal	Year
1	XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniques	KDD	2024
2	Comparing explanations in RL	Neural Comput. Appl.	2023
3	(When) Are Contrastive Explanations of Reinforcement Learning Helpful?	HiLL Workshop @ NeurIPS	2022
4	Explaining Reinforcement Learning to Mere Mortals: An Empirical Study	IJCAI	2019

Back to top

Contributing

Contributions are welcome! To suggest a paper, open an issue with the paper title, venue, year, and a link (DOI preferred).

Contact: contact (at) xrl (dot) ai

Citation

If you use this resource in your research, please cite the following paper. See CITATION.cff for citation details.

Star History