Explaining Reinforcement Learning to Mere Mortals: An Empirical Study

@article{Anderson2019ExplainingRL,
  title={Explaining Reinforcement Learning to Mere Mortals: An Empirical Study},
  author={Andrew Anderson and Jonathan Dodge and Amrita Sadarangani and Zoe Juozapaitis and Evan Newman and Jed Irvine and Souti Chattopadhyay and Alan Fern and Margaret M. Burnett},
  journal={ArXiv},
  year={2019},
  volume={abs/1903.09708}
}
We present a user study to investigate the impact of explanations on non-experts' understanding of reinforcement learning (RL) agents. [...] Key Method We designed a 124 participant, four-treatment experiment to compare participants' mental models of an RL agent in a simple Real-Time Strategy (RTS) game. Our results show that the combination of both saliency and reward bars were needed to achieve a statistically significant improvement in mental model score over the control. In addition, our qualitative…Expand
Counterfactual States for Atari Agents via Generative Deep Learning
TLDR
The user study results suggest that the generated counterfactual states are useful in helping non-expert participants gain a better understanding of an agent's decision making process. Expand
Benchmarking Perturbation-based Saliency Maps for Explaining Deep Reinforcement Learning Agents
TLDR
Four perturbation-based approaches to create saliency maps for Deep Reinforcement Learning agents trained on four different Atari 2600 games are compared using three computational metrics: dependence on the learned parameters of the agent, faithfulness to the agent’s reasoning and run-time. Expand
Explaining Reward Functions to Humans for Better Human-Robot Collaboration
TLDR
A categorization of information modalities for reward explanation techniques, a suite of assessment techniques for human reward understanding, and four axes of domain complexity are introduced to understand the effectiveness of different reward explanations techniques in a variety of domains. Expand
Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
TLDR
This paper augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to, and evaluates global and local explanation methods, providing the first user study of combined local and global explanations for RL agents. Expand
After-Action Review for AI (AAR/AI)
Explainable AI (XAI) is growing in importance as AI pervades modern society, but few have studied how XAI can directly support people trying to assess an AI agent. Without a rigorous process, peopleExpand
Human-Autonomy Teaming and Explainable AI Capabilities in RTS Games
TLDR
This work describes how it takes human-autonomy teaming and explainable artificial intelligence techniques to shed light and provide inside for players to understand recommendations by the system. Expand
Keeping it "organized and logical": after-action review for AI (AAR/AI)
TLDR
This work derived an AAR for AI, to organize ways people assess reinforcement learning agents in a sequential decision-making environment, and revealed several strengths and weaknesses of the AAR/AI process and the explanations embedded within it. Expand
Measure Utility, Gain Trust: Practical Advice for XAI Researchers
TLDR
This work outlines five broad use cases where explanations are useful and describes pseudo-experiments that rely on objective empirical measurements and falsifiable hypotheses that are necessary to contribute to scientific knowledge in the field of XAI. Expand
Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition
TLDR
This paper compares variations in explanation veracity for a controlled user study with an explainable activity recognition system and suggests that low veracity explanations significantly decrease user performance and agreement compared to both accurate explanations and a system without explanations. Expand
Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Black Box Simulators
TLDR
Methods for providing contrastive explanations in terms of user-specified concepts for sequential decision-making settings where the system's model of the task may be best represented as a blackbox simulator are introduced. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 45 REFERENCES
Visualizing and Understanding Atari Agents
TLDR
A method for generating useful saliency maps is introduced and used to show 1) what strong agents attend to, 2) whether agents are making decisions for the right or wrong reasons, and 3) how agents evolve during learning. Expand
Toward Foraging for Understanding of StarCraft Agents: An Empirical Study
TLDR
A formative study into how experienced users foraged in the domain of StarCraft to assess an agent showed that participants faced difficult foraging problems. Expand
How the Experts Do It: Assessing and Explaining Agent Behaviors in Real-Time Strategy Games
TLDR
Insight is provided into shoutcasters' foraging strategies for gleaning information necessary to assess and explain the players; a characterization of the types of implicit questions shoutcasters answered; and implications for creating explanations by using the patterns and abstraction levels these human experts revealed. Expand
Hybrid Reward Architecture for Reinforcement Learning
TLDR
A new method is proposed, called Hybrid Reward Architecture (HRA), which takes as input a decomposed reward function and learns a separate value function for each component reward function, enabling more effective learning. Expand
Metrics for Explainable AI: Challenges and Prospects
TLDR
This paper discusses specific methods for evaluating the goodness of explanations, whether users are satisfied by explanations, how well users understand the AI systems, and how the human-XAI work system performs. Expand
Interpretable Explanations of Black Boxes by Meaningful Perturbation
  • Ruth C. Fong, A. Vedaldi
  • Computer Science, Mathematics
  • 2017 IEEE International Conference on Computer Vision (ICCV)
  • 2017
TLDR
A general framework for learning different kinds of explanations for any black box algorithm is proposed and the framework to find the part of an image most responsible for a classifier decision is specialised. Expand
Sanity Checks for Saliency Maps
TLDR
It is shown that some existing saliency methods are independent both of the model and of the data generating process, and methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model. Expand
Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games
TLDR
The results show that spectators have a remarkable ability to infer intent accurately using all nine visualizations, but that visualizations with certain characteristics were more comprehensible and more readily revealed the player's intent. Expand
Q-Decomposition for Reinforcement Learning Agents
The paper explores a very simple agent design method called Q-decomposition, wherein a complex agent is built from simpler subagents. Each subagent has its own reward function and runs its ownExpand
A unified view of gradient-based attribution methods for Deep Neural Networks
TLDR
This work analyzes various state-of-the-art attribution methods and proves unexplored connections between them, and performs an empirical evaluation with six attribution methods on a variety of tasks and architectures. Expand
...
1
2
3
4
5
...