On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

  title={On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation},
  author={Sebastian Bach and Alexander Binder and Gr{\'e}goire Montavon and Frederick Klauschen and Klaus-Robert M{\"u}ller and Wojciech Samek},
  journal={PLoS ONE},
Understanding and interpreting classification decisions of automated image classification systems is of high value in many applications, as it allows to verify the reasoning of the system and provides additional information to the human expert. [] Key Method We introduce a methodology that allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over Bag of Words features and for multilayered neural networks.

Opening the machine learning black box with Layer-wise Relevance Propagation

This thesis describes a novel method for explaining non-linear classifier decisions by decomposing the prediction function, called Layer-wise Relevance Propagation (LRP), and applies this method to Neural Networks, kernelized Support Vector Machines and Bag of Words feature extraction pipelines.

Analyzing Classifiers: Fisher Vectors and Deep Neural Networks

This paper extends the LRP framework for Layer-wise Relevance Propagation for Fisher vector classifiers and uses it as analysis tool to quantify the importance of context for classification, qualitatively compare DNNs against FV classifiers in terms of important image regions and detect potential flaws and biases in data.

Validation and generalization of pixel-wise relevance in convolutional neural networks trained for face classification

Fine-grained analyses of relevance maps across models revealed asymmetries in generalization that point to specific benefits of choice parameters, and suggest that it may be possible to find an underlying set of important face image pixels that drive decisions across convolutional neural networks and tasks.

Review of white box methods for explanations of convolutional neural networks in image classification tasks

This work aims to provide a comprehensive and detailed overview of a set of methods that can be used to create explanation maps for a particular image, which assign an importance score to each pixel of the image based on its contribution to the decision of the network.

Transforming Convolutional Neural Network to an Interpretable Classifier

An effective way of transforming a sufficiently well trained convolutional neural network to an interpretable classifier is presented, able not only to predict the class of an unknown observation, but also to justify the prediction by providing three most similar training observations in exchange to a slightly decreased accuracy.


The pioneering work of using visualization to improve the explainability of a classifier at different stages in the life circle is discussed, and the challenges and future research opportunities are discussed.

Contextual Prediction Difference Analysis for Explaining Individual Image Classifications

This work first shows that PDA can suffer from saturated classifiers, then proposes Contextual PDA, which runs hundreds of times faster than PDA and is shown to be superior by explaining image classifications of the state-of-the-art deep convolutional neural networks.

Explaining nonlinear classification decisions with deep Taylor decomposition

Explaining Predictions of Deep Neural Classifier via Activation Analysis

A novel approach to explain and support an interpretation of the decision-making process to a human expert operating a deep learning system based on Convolutional Neural Network (CNN).

Understanding Individual Decisions of CNNs via Contrastive Backpropagation

Contrastive Layer-wise Relevance Propagation is proposed, which is capable of producing instance-specific, class-discriminative, pixel-wise explanations and both qualitative and quantitative evaluations show that the CLRP generates better explanations than the LRP.



What has my classifier learned? Visualizing the classification rules of bag-of-feature model by support region detection

  • Lingqiao LiuLei Wang
  • Computer Science
    2012 IEEE Conference on Computer Vision and Pattern Recognition
  • 2012
This work developed an efficient RSRS detection algorithm that showed that it can be used to identify the limitation of a classifier, predict its failure mode, discover the classification rules and reveal the database bias.

On Taxonomies for Multi-class Image Categorization

Empirical results show that the local SVM formulation can effectively exploit the taxonomy structure and thus outperforms standard multi-class classification algorithms while it achieves on par results with taxonomy-based structured algorithms at a significantly decreased computing time.

Insights from Classifying Visual Concepts with Multiple Kernel Learning

A recently developed non-sparse MKL variant is applied to state-of-the-art concept recognition tasks from the application domain of computer vision and compared against its direct competitors, the sum-kernel SVM and sparse MKL.

How to Explain Individual Classification Decisions

This paper proposes a procedure which (based on a set of assumptions) allows to explain the decisions of any classification method.

Improving the Fisher Kernel for Large-Scale Image Classification

In an evaluation involving hundreds of thousands of training images, it is shown that classifiers learned on Flickr groups perform surprisingly well and that they can complement classifier learned on more carefully annotated datasets.

Visualizing and Understanding Convolutional Networks

A novel visualization technique is introduced that gives insight into the function of intermediate feature layers and the operation of the classifier in large Convolutional Network models, used in a diagnostic role to find model architectures that outperform Krizhevsky et al on the ImageNet classification benchmark.

Neural Networks: Tricks of the Trade

It is shown how nonlinear semi-supervised embedding algorithms popular for use with â œshallowâ learning techniques such as kernel methods can be easily applied to deep multi-layer architectures.

Visualization of Nonlinear Classification Models in Neuroimaging - Signed Sensitivity Maps

This work focuses on the generation of summary maps of a nonlinear classifier, that reveal how the classifier works in different parts of the input domain, unlike earlier related methods.

Image Classification with the Fisher Vector: Theory and Practice

This work proposes to use the Fisher Kernel framework as an alternative patch encoding strategy: it describes patches by their deviation from an “universal” generative Gaussian mixture model, and reports experimental results showing that the FV framework is a state-of-the-art patch encoding technique.