• Corpus ID: 236134448

Adaptive wavelet distillation from neural networks through interpretations

@inproceedings{Ha2021AdaptiveWD,
  title={Adaptive wavelet distillation from neural networks through interpretations},
  author={Wooseok Ha and Chandan Singh and François Lanusse and Eli Song and Song Dang and Kangmin He and Srigokul Upadhyayula and Bin Yu},
  booktitle={NeurIPS},
  year={2021}
}
Recent deep-learning models have achieved impressive prediction performance, but often sacrifice interpretability and computational efficiency. Interpretability is crucial in many disciplines, such as science and medicine, where models must be carefully vetted or where interpretation is the goal itself. Moreover, interpretable models are concise and often yield computational efficiency. Here, we propose adaptive wavelet distillation (AWD), a method which aims to distill information from a trained… 

Figures and Tables from this paper

Interpreting and improving deep-learning models with reality checks
TLDR
This chapter covers recent work aiming to interpret models by attributing importance to features and feature groups for a single prediction, and shows how these attributions can be used to directly improve the generalization of a neural network or to distill it into a simple model.
Fast Interpretable Greedy-Tree Sums (FIGS)
TLDR
FIGS generalizes the CART algorithm to simultaneously grow a flexible number of trees in a summation, and is able to avoid repeated splits, and often provides more concise decision rules than fitted decision trees, without sacrificing predictive performance.
Robust Time Series Denoising with Learnable Wavelet Packet Transform
TLDR
This paper proposes to apply a deep learning denoising model inspired by a signal processing, a learnable version of wavelet packet transform that has signficant learning capabilities with few interpretable parameters and has an intuitive initialisation.
Wavelet-Packets for Deepfake Image Analysis and Detection
TLDR
This paper proposes to learn a model for the detection of synthetic images based on the wavelet-packet representation of natural and generative adversarial neural network (GAN)-generated images, and studies the binary Face Forensics++ (ff++) fake-detection problem.
Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow
TLDR
Novel analyses and a new framework are presented that explain the physics learned in TL and provide a framework to guide TL for wide-ranging applications in science and engineering, such as climate change modeling.
Machine Learning and Cosmology
TLDR
Current and ongoing developments relating to the application of machine learning within cosmology and a set of recommendations aimed at maximizing the scientific impact of these burgeoning tools over the coming decade through both technical development as well as the fostering of emerging communities are summarized.
Learnable Wavelet Packet Transform for Data-Adapted Spectrograms
TLDR
A deep learning framework for learnable wavelet packet transforms is proposed, enabling to learn features automatically from data and optimise them with respect to the defined objective function.
Group Probability-Weighted Tree Sums for Interpretable Modeling of Heterogeneous Data
TLDR
An instance-weighted tree-sum method that effectively pools data across diverse groups to output a concise, rule-based model that achieves state-of-the-art prediction performance on important clinical datasets.
Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement
TLDR
A comprehensive overview over techniques that apply XAI practically for improving various properties of ML models, and systematically categorizes these approaches, comparing their respective strengths and weaknesses is provided.
Inferring dark matter substructure with astrometric lensing beyond the power spectrum
TLDR
This work introduces a novel method to search for global dark matter-induced gravitational lensing signatures in astrometric datasets based on neural likelihood-ratio estimation, showing significantly enhanced sensitivity to a cold dark matter population and more favorable scaling with measurement noise compared to existing approaches.
...
...

References

SHOWING 1-10 OF 72 REFERENCES
Transformation Importance with Applications to Cosmology
TLDR
The proposed TRIM (TRansformation IMportance), a novel approach which attributes importances to features in a transformed space and can be applied post-hoc to a fully trained model, is motivated by a cosmological parameter estimation problem using deep neural networks on simulated data.
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges
TLDR
This work provides fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic.
Hierarchical interpretations for neural network predictions
TLDR
This work introduces the use of hierarchical interpretations to explain DNN predictions through the proposed method, agglomerative contextual decomposition (ACD), and demonstrates that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs.
A Unified Approach to Interpreting Model Predictions
TLDR
A unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations), which unifies six existing methods and presents new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.
WaveletKernelNet: An Interpretable Deep Neural Network for Industrial Intelligent Diagnosis
TLDR
A novel wavelet-driven deep neural network, termed as WaveletKernelNet (WKN), is presented, where a continuous wavelet convolutional (CWConv) layer is designed to replace the first convolutionAL layer of the standard CNN.
Interpretable Machine Learning
TLDR
This project introduces Robust T CAV, which builds on TCAV and experimentally determines best practices for this method and is a step in the direction of making TCAVs, an already impactful algorithm in interpretability, more reliable and useful for practitioners.
Learning Global Additive Explanations for Neural Nets Using Model Distillation
TLDR
This work proposes to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions, taking the form of feature shapes, which are more expressive than feature attributions.
Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
For an explanation of a deep learning model to be effective, it must provide both insight into a model and suggest a corresponding action in order to achieve some objective. Too often, the litany of
Definitions, methods, and applications in interpretable machine learning
TLDR
This work defines interpretability in the context of machine learning and introduces the predictive, descriptive, relevant (PDR) framework for discussing interpretations, and introduces 3 overarching desiderata for evaluation: predictive accuracy, descriptive accuracy, and relevancy.
TNT: An Interpretable Tree-Network-Tree Learning Framework using Knowledge Distillation
TLDR
A Tree-Network-Tree (TNT) learning framework for explainable decision-making, where the knowledge is alternately transferred between the tree model and DNNs is proposed, and extensive experiments demonstrated the effectiveness of the proposed method.
...
...