ALICE: Active Learning with Contrastive Natural Language Explanations

@inproceedings{Liang2020ALICEAL,
  title={ALICE: Active Learning with Contrastive Natural Language Explanations},
  author={Weixin Liang and James Y. Zou and Zhou Yu},
  booktitle={EMNLP},
  year={2020}
}
Training a supervised neural network classifier typically requires many annotated training samples. Collecting and annotating a large number of data points are costly and sometimes even infeasible. Traditional annotation process uses a low-bandwidth human-machine communication interface: classification labels, each of which only provides several bits of information. We propose Active Learning with Contrastive Explanations (ALICE), an expert-in-the-loop training framework that utilizes… Expand

Figures and Tables from this paper

Teach me how to Label: Labeling Functions from Natural Language with Text-to-text Transformers
TLDR
The task of turning natural language descriptions into Python labeling functions by following a novel approach to semantic parsing with pre-trained text-to-text Transformers achieves a new state of the art on the semantic parsing benchmark CoNaLa, surpassing the previous best approach by 3.7 BLEU points. Expand
Learning Transferable Visual Models From Natural Language Supervision
TLDR
It is demonstrated that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. Expand
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
TLDR
This paper gives a formal framework for the available modeling approaches, in which explanation data can be used as model inputs, as targets, or as a prior, and proposes to use a retrieval-based method that solves the authors' synthetic task with accuracies upwards of 95%, while baselines without explanation data achieve below 65% accuracy. Expand
LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering
TLDR
LRTA [Look, Read, Think, Answer], a transparent neural-symbolic reasoning framework for visual question answering that solves the problem step-by-step like humans and provides human-readable form of justification at each step is proposed. Expand
HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations
TLDR
This work proposes HERALD, an annotation efficient framework that reframes the training data annotation process as a denoising problem, and shows that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora. Expand
A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification
TLDR
A general real-world AL strategy is introduced as part of a learning cycle and its elements are used, e.g., the query and annotator selection algorithm, to categorize about 60 real- world AL strategies. Expand
Leveraging Language to Learn Program Abstractions and Search Heuristics
TLDR
This work introduces LAPS (Language for Abstraction and Program Search), a technique for using natural language annotations to guide joint learning of libraries and neurally-guided search models for synthesis. Expand
Learning to Recognize Dialect Features
TLDR
Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples. Expand
GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering
TLDR
This work proposes GraphVQA, a language-guided graph neural network framework that translates and executes a natural language question as multiple iterations of message passing among graph nodes, and explores the design space of the framework, and discusses the trade-off of different design choices. Expand
Neural Group Testing to Accelerate Deep Learning
TLDR
Inspired by the group testing strategy for efficient disease testing, neural group testing is proposed, which accelerates by testing a group of samples in one forward pass and can reduce the overall computation cost by over 73% while improving detection performance. Expand
...
1
2
...

References

SHOWING 1-10 OF 49 REFERENCES
Joint Concept Learning and Semantic Parsing from Natural Language Explanations
TLDR
A joint model for language interpretation (semantic parsing) and concept learning (classification) that does not require labeling statements with logical forms is presented, which prefers discriminative interpretations of statements in context of observable features of the data as a weak signal for parsing. Expand
Learning with Latent Language
TLDR
This paper aims to show that using the space of natural language strings as a parameter space is an effective way to capture natural task structure and shows that, in all settings, models with a linguistic parameterization outperform those without. Expand
Training Classifiers with Natural Language Explanations
TLDR
BabbleLabble is proposed, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision, and it is found that users are able to train classifiers with comparable F1 scores from 5-100 faster by providing explanations instead of just labels. Expand
Modeling Annotators: A Generative Approach to Learning from Annotator Rationales
TLDR
A generative model of how a given annotator, knowing the true θ, stochastically chooses rationales is presented, and observing the rationales helps us infer thetrue θ. Expand
NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extraction
TLDR
A neural approach to ground rules for RE is presented, named Nero, which jointly learns a relation extraction module and a soft matching module that learns to match rules with semantically similar sentences such that raw corpora can be automatically labeled and leveraged by the RE module (in a much better coverage) as augmented supervision. Expand
Generating Visual Explanations
TLDR
A new model is proposed that focuses on the discriminating properties of the visible object, jointly predicts a class label, and explains why the predicted label is appropriate for the image, and generates sentences that realize a global sentence property, such as class specificity. Expand
Active Learning Literature Survey
TLDR
This report provides a general introduction to active learning and a survey of the literature, including a discussion of the scenarios in which queries can be formulated, and an overview of the query strategy frameworks proposed in the literature to date. Expand
ExpBERT: Representation Engineering with Natural Language Explanations
TLDR
Across three relation extraction tasks, the method, ExpBERT, matches a BERT baseline but with 3–20x less labeled data and improves on the baseline by 3–10 F1 points with the same amount of labeled data. Expand
Group-Based Active Learning of Classification Models
TLDR
A new group-based active learning framework is developed that solicits label information on groups of multiple examples in order to describe groups in a user-friendly way, and conjunctive patterns are used to compactly represent groups. Expand
Explaining Explanations in AI
TLDR
This work contrasts the different schools of thought on what makes an explanation in philosophy and sociology, and suggests that machine learning might benefit from viewing the problem more broadly. Expand
...
1
2
3
4
5
...