Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning

  title={Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning},
  author={Rongzhi Zhang and Yue Yu and Pranav Shetty and Le Song and Chao Zhang},
Weakly-supervised learning (WSL) has shown promising results in addressing label scarcity on many NLP tasks, but manually designing a comprehensive, high-quality labeling rule set is tedious and difficult. We study interactive weakly-supervised learning—the problem of iteratively and automatically discovering novel labeling rules from data to improve the WSL model. Our proposed model, named PRBoost, achieves this goal via iterative prompt-based rule discovery and model boosting. It uses… 

Figures and Tables from this paper

Automatic Rule Induction for Efficient Semi-Supervised Learning
Automatic Rule Induction is proposed, a simple and general-purpose framework for the automatic discovery and integration of symbolic rules into pretrained transformer models that can improve state-of-the-art methods with no manual effort and minimal computational over-head.
AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Models
Experiments show that A C T UNE outperforms the strongest active learning and self-training baselines and improves the label efficiency of PLM finetuning by 56.2% on average.
Language Models in the Loop: Incorporating Prompting into Weak Supervision
The experimental evaluation shows that prompting large language models within a weak supervision framework can provide gains in accuracy, and that this approach produces classifiers with comparable or superior accuracy to those trained from hand-engineered rules.
Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming
Nemo is presented, an end-to-end interactive system that improves the overall productivity of WS learning pipeline by an average 20% (and up to 47% in one task) compared to the prevailing WS approach.
Understanding Programmatic Weak Supervision via Source-aware Influence Function
This work builds on Influence Function (IF) and proposes source-aware IF, which leverages the generation process of the probabilistic labels to decompose the end model's training objective and then calculates the influence associated with each data, source, class tuple.


GLaRA: Graph-based Labeling Rule Augmentation for Weakly Supervised Named Entity Recognition
This work proposes GLARA, a graph-based labeling rule augmentation framework, to learn new labeling rules from unlabeled data, and designs a new graph neural network to augment labeling rules by exploring the semantic relations between rules.
Weakly Supervised Named Entity Tagging with Learnable Logical Rules
This work proposes a novel method TALLOR that bootstraps high-quality logical rules to train a neural tagger in a fully automated manner that outperforms other weakly supervised methods and rivals a state-of-the-art distantly supervised tagger with a lexicon of over 2,000 terms when starting from only 20 simple rules.
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling
This work develops the first framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic, demonstrating that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels.
Adaptive Rule Discovery for Labeling Text Data
DARWIN, an interactive system designed to alleviate the task of writing rules for labeling text data in weakly-supervised settings, is presented and it is demonstrated that rules discovered by DARWIN on average identify 40% more positive instances compared to Snuba even when it is provided with 1000 labeled instances.
NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extraction
A neural approach to ground rules for RE is presented, named Nero, which jointly learns a relation extraction module and a soft matching module that learns to match rules with semantically similar sentences such that raw corpora can be automatically labeled and leveraged by the RE module (in a much better coverage) as augmented supervision.
Learning from Rules Generalizing Labeled Exemplars
A training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables, shows that it is more accurate than several existing methods of learning from a mix of clean and noisy supervision.
TagRuler: Interactive Tool for Span-Level Data Programming by Demonstration
A novel tool is built, TagRuler, that makes it easy for annotators to build span-level labeling functions without programming and encourages them to explore trade-offs between different labeling models and active learning strategies.
Weakly Supervised Sequence Tagging from Noisy Rules
A new type of generative model, linked hidden Markov models (linked HMMs), are introduced and it is found that linked HMMs provide an average 7 F1 point boost on benchmark named entity recognition tasks versus generative models that assume the tags are i.i.d.
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
This work introduces Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.
ATM: An Uncertainty-aware Active Self-training Framework for Label-efficient Text Classification
ATM is a new framework that leverages self-training to exploit unlabeled data and is agnostic to the specific AL algorithm, serving as a plug-in module to improve existing AL methods.