Understanding Programmatic Weak Supervision via Source-aware Influence Function

  title={Understanding Programmatic Weak Supervision via Source-aware Influence Function},
  author={Jieyu Zhang and Hong Wang and Cheng-Yu Hsieh and Alexander J. Ratner},
Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end model. With its increasing popularity, it is critical to have some tool for users to understand the influence of each component ( e.g. , the source vote or training data) in the pipeline and interpret the end model behavior. To achieve this, we build on Influence Function (IF) and propose source-aware IF, which leverages… 

Figures and Tables from this paper

Binary Classification with Positive Labeling Sources
This work proposes Weapo, a simple yet competitive WS method for producing training labels without negative labeling sources, and shows Weapo achieves the highest averaged performance in terms of both the quality of synthesized labels and the performance of the classifier supervised with these labels.
Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction
AMRule is developed, a multi-view rule discovery framework that can adaptively and iteratively discover novel rulers that can complement the current weakly-supervised model to improve compatibility prediction and improves rule quality and rule proposal efficiency.


RelatIF: Identifying Explanatory Training Examples via Relative Influence
RelatIF is introduced, a new class of criteria for choosing relevant training examples by way of an optimization objective that places a constraint on global influence and finds that the examples returned are more intuitive when compared to those found using influence functions.
Training Complex Models with Multi-Task Weak Supervision
This work shows that by solving a matrix completion-style problem, it can recover the accuracies of these multi-task sources given their dependency structure, but without any labeled data, leading to higher-quality supervision for training an end model.
On the Accuracy of Influence Functions for Measuring Group Effects
Across many different types of groups and for a range of real-world datasets, the predicted effect (using influence functions) of a group correlates surprisingly well with its actual effect, even if the absolute and relative errors are large.
WRENCH: A Comprehensive Benchmark for Weak Supervision
A benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches, consisting of 22 varied real-world datasets for classification and sequence tagging; a range of real, synthetic, and procedurally-generated weak supervision sources; and a modular, extensible framework for WS evaluation, including implementations for popular WS methods.
Data Programming: Creating Large Training Sets, Quickly
A paradigm for the programmatic creation of training sets called data programming is proposed in which users express weak supervision strategies or domain heuristics as labeling functions, which are programs that label subsets of the data, but that are noisy and may conflict.
Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm
The EM algorithm is shown to provide a slow but sure way of obtaining maximum likelihood estimates of the parameters of interest in compiling a patient record.
Understanding Black-box Predictions via Influence Functions
This paper uses influence functions — a classic technique from robust statistics — to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction.
Resolving training biases via influencebased data relabeling
  • In International Conference on Learning Representations,
  • 2022
Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning
This proposed model, named PRBoost, uses boosting to identify large-error instances and discovers candidate rules from them by prompting pre-trained LMs with rule templates, and bridges the gaps with fully supervised models.
Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming
Nemo is presented, an end-to-end interactive system that improves the overall productivity of WS learning pipeline by an average 20% (and up to 47% in one task) compared to the prevailing WS approach.