• Corpus ID: 240354726

On Quantitative Evaluations of Counterfactuals

  title={On Quantitative Evaluations of Counterfactuals},
  author={Frederik Hvilsh{\o}j and Alexandros Iosifidis and Ira Assent},
As counterfactual examples become increasingly popular for explaining decisions of deep learning models, it is essential to understand what properties quantitative evaluation metrics do capture and equally important what they do not capture. Currently, such understanding is lacking, potentially slowing down scientific progress. In this paper, we consolidate the work on evaluating visual counterfactual examples through an analysis and experiments. We find that while most metrics behave as… 

Figures and Tables from this paper

Diffusion Models for Counterfactual Explanations
DiME, a method allowing the generation of counterfactual images using the recent diffusion models by leveraging the guided generative diffusion process, is proposed, which surpasses previous State-of-the-Art results on 5 out of 6 metrics on CelebA.
Diffeomorphic Counterfactuals with Generative Models
This work performs a suitable diffeomorphic coordinate transformation and performs gradient ascent in these coordinates to generate counterfactuals which are classed with great confidence as a speci⬁ed target class.
Fighting Money Laundering with Statistics and Machine Learning: An Introduction and Review
This paper proposes a unifying terminology and provides a review of the literature on anti-money laundering in banks to help organize existing research, and discusses directions for future research.
A survey of algorithmic recourse:contrastive explanations and consequential recommendations
This work focuses on algorithmic recourse, which is concerned with providing explanations and recommendations to individuals who are unfavorably treated by automated decision-making systems, and performs an extensive literature review.


Explaining machine learning classifiers through diverse counterfactual explanations
This work proposes a framework for generating and evaluating a diverse set of counterfactual explanations based on determinantal point processes, and provides metrics that enable comparison ofcounterfactual-based methods to other local explanation methods.
Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations
DiVE learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss to uncover multiple valuable explanations about the model’s prediction and introduces a mechanism to prevent the model from producing trivial explanations.
Learning Model-Agnostic Counterfactual Explanations for Tabular Data
A framework, called C-CHVAE, is developed, drawing ideas from the manifold learning literature, that generates faithful counterfactuals and is suggested to complement the catalog ofcounterfactual quality measures using a criterion to quantify the degree of difficulty for a certainCounterfactual suggestion.
ECINN: Efficient Counterfactuals from Invertible Neural Networks
This work proposes a method, ECINN, that utilizes the generative capacities of invertible neural networks for image classification to generate counterfactual examples e.g. in the time of only two evaluations, in contrast to competing methods that sometimes need a thousand evaluations or more.
Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers
The problem of feasibility is formulated as preserving causal relationships among input features and a method is presented that uses (partial) structural causal models to generate actionable counterfactuals that better satisfy feasibility constraints than existing methods.
Counterfactual Explanation Based on Gradual Construction for Deep Networks
Generating Interpretable Counterfactual Explanations By Implicit Minimisation of Epistemic and Aleatoric Uncertainties
This work introduces a simple and fast method for generating interpretable CEs in a white-box setting without an auxiliary model, by using the predictive uncertainty of the classifier.
A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence
This work conducts a systematic literature review which provides readers with a thorough and reproducible analysis of the interdisciplinary research field under study and defines a taxonomy regarding both theoretical and practical approaches to contrastive and counterfactual explanation.
Interpretable Counterfactual Explanations Guided by Prototypes
It is shown that class prototypes, obtained using either an encoder or through class specific k-d trees, significantly speed up the search for counterfactual instances and result in more interpretable explanations.
Explanation by Progressive Exaggeration
This work proposes a method that explains the outcome of a classification black-box by gradually exaggerating the semantic effect of a given class by gradually changes the posterior probability from its original class to its negation.