• Corpus ID: 246441887

Active Learning Over Multiple Domains in Natural Language Tasks

  title={Active Learning Over Multiple Domains in Natural Language Tasks},
  author={S. Longpre and Julia Reisler and Edward Greg Huang and Yi Lu and Andrew J. Frank and Nikhil Ramesh and Chris DuBois},
Studies of active learning traditionally assume the target and source data stem from a single domain. However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear a priori which data sources will help or hurt the target domain. We survey a wide variety of techniques in active learning (AL), domain shift detection (DS), and multi-domain sampling to examine this challenging setting for question answering and… 


Strong Baselines for Neural Semi-Supervised Learning under Domain Shift
This paper re-evaluate classic general-purpose bootstrapping approaches in the context of neural networks under domain shifts vs. recent neural approaches and proposes a novel multi-task tri-training method that reduces the time and space complexity of classic tri- training.
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
This work investigates the relative benefits of large pre-trained language models, various data sampling strategies, as well as query and context paraphrases generated by back-translation, and finds a simple negative sampling technique to be particularly effective.
Active Learning for BERT: An Empirical Study
The results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class.
A Review of Domain Adaptation without Target Labels
  • Wouter M. Kouw, M. Loog
  • Computer Science
    IEEE Transactions on Pattern Analysis and Machine Intelligence
  • 2021
Domain adaptation has become a prominent problem setting in machine learning and related fields. This review asks the question: How can a classifier learn from a source domain and generalize to a
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale
The best zero-shot transfer model considerably outperforms in-domain BERT and the previous state of the art on six benchmarks, and is proposed to incorporate self-supervised with supervised multi-task learning on all available source domains.
Selective Question Answering under Domain Shift
This work proposes the setting of selective question answering under domain shift, in which a QA model is tested on a mixture of in-domain and out-of-domain data, and must answer as many questions as possible while maintaining high accuracy.
A theory of learning from different domains
A classifier-induced divergence measure that can be estimated from finite, unlabeled samples from the domains and shows how to choose the optimal combination of source and target error as a function of the divergence, the sample sizes of both domains, and the complexity of the hypothesis class.
To Annotate or Not? Predicting Performance Drop under Domain Shift
This paper investigates three families of methods (\mathcal{H}-divergence, reverse classification accuracy and confidence measures), shows how they can be used to predict the performance drop and study their robustness to adversarial domain-shifts.
Practical Obstacles to Deploying Active Learning
It is shown that while AL may provide benefits when used with specific models and for particular domains, the benefits of current approaches do not generalize reliably across models and tasks.
Analysis of Representations for Domain Adaptation
The theory illustrates the tradeoffs inherent in designing a representation for domain adaptation and gives a new justification for a recently proposed model which explicitly minimizes the difference between the source and target domains, while at the same time maximizing the margin of the training set.