Emb-GAM: an Interpretable and Efficient Predictor using Pre-trained Language Models

  title={Emb-GAM: an Interpretable and Efficient Predictor using Pre-trained Language Models},
  author={Chandan Singh and Jianfeng Gao},
Deep learning models have achieved impressive prediction performance but often sacrifice interpretability and speed, critical considerations in high-stakes domains and compute-limited settings. In contrast, generalized additive models (GAMs) can maintain interpretability and speed but often suffer from poor prediction performance due to their inability to effectively capture feature interactions. This work aims to bridge this gap by using pre-trained neural language models to extract embeddings… 

Figures and Tables from this paper

iPrompt: Explaining Data Patterns in Natural Language via Interpretable Autoprompting

Inter interpretable autoprompting ( iPrompt ), an algorithm that generates a natural language string explaining the data that is human-interpretable and able to aid in scientific discovery.

Explaining Patterns in Data with Language Models via Interpretable Autoprompting

Experiments on a wide range of datasets show that iPrompt can yield meaningful insights by accurately identifying groundtruth dataset descriptions and producing prompts that improve upon prompts for generalization.



DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

This work proposes a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can be fine-tuned with good performances on a wide range of tasks like its larger counterparts, and introduces a triple loss combining language modeling, distillation and cosine-distance losses.

Hierarchical interpretations for neural network predictions

This work introduces the use of hierarchical interpretations to explain DNN predictions through the proposed method, agglomerative contextual decomposition (ACD), and demonstrates that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs.

Neural Additive Models: Interpretable Machine Learning with Neural Nets

Neural Additive Models (NAMs) are proposed which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models and are more accurate than widely used intelligible models such as logistic regression and shallow decision trees.

Deep Learning for Case-based Reasoning through Prototypes: A Neural Network that Explains its Predictions

This work creates a novel network architecture for deep learning that naturally explains its own reasoning for each prediction, and the explanations are loyal to what the network actually computes.

Adaptive wavelet distillation from neural networks through interpretations

Adapt wavelet distillation (AWD), a method which aims to distill information from a trained neural network into a wavelet transform, which gives predictive performance better than state-of-the-art neural networks and has properties which make it easy to interpret.

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

This work provides fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic.

Transformation Importance with Applications to Cosmology

The proposed TRIM (TRansformation IMportance), a novel approach which attributes importances to features in a transformed space and can be applied post-hoc to a fully trained model, is motivated by a cosmological parameter estimation problem using deep neural networks on simulated data.

Definitions, methods, and applications in interpretable machine learning

This work defines interpretability in the context of machine learning and introduces the predictive, descriptive, relevant (PDR) framework for discussing interpretations, and introduces 3 overarching desiderata for evaluation: predictive accuracy, descriptive accuracy, and relevancy.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

It is found that BERT was significantly undertrained, and can match or exceed the performance of every model published after it, and the best model achieves state-of-the-art results on GLUE, RACE and SQuAD.

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.