Why? Why not? When? Visual Explanations of Agent Behaviour in Reinforcement Learning

  title={Why? Why not? When? Visual Explanations of Agent Behaviour in Reinforcement Learning},
  author={Aditi Mishra and Utkarsh Soni and Jinbin Huang and Chris Bryan},
  journal={2022 IEEE 15th Pacific Visualization Symposium (PacificVis)},
Reinforcement learning (RL) is used in many domains, including autonomous driving, robotics, stock trading, and video games. Unfortunately, the black box nature of RL agents, combined with legal and ethical considerations, makes it increasingly important that humans (including those are who not experts in RL) understand the reasoning behind the actions taken by an RL agent, particularly in safety-critical domains. To help address this challenge, we introduce PolicyExplainer, a visual analytics… 

Figures from this paper

“Why did my AI agent lose?”: Visual Analytics for Scaling Up After-Action Review

This paper augments the AAR/AI process to be performed at three levels—episode- level, decision-level, and explanation-level—and integrates it into the authors' redesigned visual analytics interface to help visualization play a more critical role in AI interpretability.

Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RL

This paper considers testing RL agents that make decisions via online tree search using a learned transition model and value function and presents a user study involving knowledgeable AI researchers using the recent CheckList testing methodology to evaluate an agent trained to play a complex real-time strategy game.

Explainability in Deep Reinforcement Learning, a Review into Current Methods and Applications

The use of Deep Reinforcement Learning schemes has increased dramatically since their introduction in 2015 and a review looks at which methods are being used and what applications they are being using to identify which models are the best suited to each application or if a method is being underutilised.

ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspective.

ConCEPTEXPLAINER, a visual analytics system that enables people to interactively probe and explore the concept space to explain model behavior at the instance/class/global level, is developed and validated to address a number of design challenges that model users face in interpreting the behavior of deep learning models.

Observing and Understanding Agent’s Characteristics With Environmental Changes for Learning Agents

This paper has presented a card game that would enable the analysis of learning agents’ characteristics with environmental changes to be enhanced and provide more space or knowledge about how to improve the agent.



Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

This study proposes a method that enables a RL agent to explain its behavior in terms of the expected consequences of state transitions and outcomes, and developed a procedure that enables the agent to obtain the consequences of a single action, as well as its entire policy.

Visualizing and Understanding Atari Agents

A method for generating useful saliency maps is introduced and used to show 1) what strong agents attend to, 2) whether agents are making decisions for the right or wrong reasons, and 3) how agents evolve during learning.

DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning

We present DRLViz, a visual analytics interface to interpret the internal memory of an agent (e.g. a robot) trained using deep reinforcement learning. This memory is composed of large temporal

Improving Robot Controller Transparency Through Autonomous Policy Explanation

  • Bradley HayesJ. Shah
  • Computer Science
    2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI
  • 2017
This work presents a series of algorithms and an accompanying system that enables robots to autonomously synthesize policy descriptions and respond to both general and targeted queries by human collaborators, demonstrating applicability to a variety of robot controller types.

The Societal Implications of Deep Reinforcement Learning

Recent progress in DRL is reviewed, how this may introduce novel and pressing issues for society, ethics, and governance are discussed, and important avenues for future research are highlighted to better understand DRL’s societal implications.

Learn to Interpret Atari Agents

Deep Reinforcement Learning (DeepRL) agents surpass human-level performances in a multitude of tasks. However, the direct mapping from states to actions makes it hard to interpret the rationale

Distal Explanations for Explainable Reinforcement Learning Agents

A distal explanation model that can analyse counterfactuals and opportunity chains using decision trees and causal models is introduced and investigates the participants' understanding of the agent through task prediction and their subjective satisfaction of the explanations and shows that the model performs better in task prediction.

DQNViz: A Visual Analytics Approach to Understand Deep Q-Networks

This work proposes DQNViz, a visual analytics system to expose details of the blind training process in four levels, and enable users to dive into the large experience space of the DQN agent for comprehensive analysis, and demonstrates that it can effectively help domain experts to understand, diagnose, and potentially improve D QN models.

Deep Reinforcement Learning: An Overview

This work discusses core RL elements, including value function, in particular, Deep Q-Network (DQN), policy, reward, model, planning, and exploration, and important mechanisms for RL, including attention and memory, unsupervised learning, transfer learning, multi-agent RL, hierarchical RL, and learning to learn.

Explaining robot actions

A system that provides insight into robotic behaviors by enabling a robot to answer questions people pose about its actions in human-understandable terms despite the mathematical, robot-specific representation and planning system used by the robot to make its decisions and execute its actions.