Knodle: Modular Weakly Supervised Learning with PyTorch

  title={Knodle: Modular Weakly Supervised Learning with PyTorch},
  author={Anastasiia Sedova and Andreas Stephan and M. Speranskaya and Benjamin Roth},
  booktitle={Workshop on Representation Learning for NLP},
Strategies for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task or integrated with a specific model architecture. In this work, we introduce Knodle, a software framework that treats weak data annotations, deep learning models, and methods for improving weakly supervised training as separate, modular components. This modularization gives the training process access to fine-grained information such as… 

Figures and Tables from this paper

WeaNF”:" Weak Supervision with Normalizing Flows

This work explores a novel direction of generative modeling for weak supervision, where instead of modeling the output of the annotation process (the labeling function matches), it generatively model the input-side data distributions (the feature space) covered by labeling functions.

SepLL: Separating Latent Class Labels from Weak Supervision Noise

This work provides a method for learning from weak labels by separating two types of complementary information associated with the labeling functions: information related to the target label and information specific to one labeling function only.

XPASC: Measuring Generalization in Weak Supervision

A novel method, XPASC (eXPlainability-Association SCore), is introduced for measuring the generalization of a model trained with a weakly supervised dataset and it is shown that generalization and performance do not relate one-to-one, and that the highest degree of generalization does not necessarily imply the best performance.

XPASC: Measuring Generalization in Weak Supervision by Explainability and Association

The XPASC score is used to measure generalization in weakly-supervised models, an adversarial architecture intended to control the degree of generalization from the labeling functions and thus to mitigate the problem of overfitting, and results and qualitative analysis show that generalization and performance do not relate one-to-one.


This paper proposes a new approach called reinforced labeling (RL), given an unlabeled dataset and a set of LFs, that augments the LFs’ outputs to cases not covered by LFs based on similarities among samples, which can lead to higher labeling coverage for training an end classifier.

ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision

Noise reduction techniques for weak supervision based on the principle of k -fold cross-validation are in-vestigated and a new algorithm for denoising the weakly annotated data called ULF is introduced, that re-weights the allocation of LFs to classes by estimating the reliable LFs-to-classes joint matrix.

Label Augmentation with Reinforced Labeling for Weak Supervision

This paper proposes a new approach called reinforced labeling (RL), given an unlabeled dataset and a set of LFs, that augments the LFs’ outputs to cases not covered by LFs based on similarities among samples, which can lead to higher labeling coverage for training an end classifier.

KnowMAN: Weakly Supervised Multinomial Adversarial Networks

KnowMAN is proposed, an adversarial scheme that enables to control influence of signals associated with specific labeling functions and forces the network to learn representations that are invariant to those signals and to pick up other signals that are more generally associated with an output label.



Snorkel: rapid training data creation with weak supervision

Snorkel is a first-of-its-kind system that enables users to train state-of theart models without hand labeling any training data by incorporating the first end-to-end implementation of the recently proposed machine learning paradigm, data programming.

Training Convolutional Networks with Noisy Labels

An extra noise layer is introduced into the network which adapts the network outputs to match the noisy label distribution and can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks.

Confident Learning: Estimating Uncertainty in Dataset Labels

This work combines the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels, and presents a generalized CL which is provably consistent and experimentally performant.

CrossWeigh: Training Named Entity Tagger from Imperfect Annotations

This study dives deep into one of the widely-adopted NER benchmark datasets, CoNLL03 NER, and proposes a simple yet effective framework, CrossWeigh, to handle label mistakes during NER model training.

Neural Relation Extraction with Selective Attention over Instances

A sentence-level attention-based model for relation extraction that employs convolutional neural networks to embed the semantics of sentences and dynamically reduce the weights of those noisy instances.

Distant supervision for relation extraction without labeled data

This work investigates an alternative paradigm that does not require labeled corpora, avoiding the domain dependence of ACE-style algorithms, and allowing the use of corpora of any size.

Learning Whom to Trust with MACE

MACE (Multi-Annotator Competence Estimation) learns in an unsupervised fashion to identify which annotators are trustworthy and predict the correct underlying labels, and shows considerable improvements over standard baselines, both for predicted label accuracy and trustworthiness estimates.

Analysing the Noise Model Error for Realistic Noisy Label Data

The theoretical results and the corresponding experiments give insights into the factors that influence the noise model estimation like the noise distribution and the sampling technique.

Data Programming using Semi-Supervision and Subset Selection

It is argued that by not using any labelled data, data programming based approaches can yield sub-optimal performance, particularly, in cases when the labelling functions are noisy.

Learning with Weak Supervision for Email Intent Detection

An end-to-end robust deep neural network model for email intent identification that leverages both clean annotated data and noisy weak supervision along with a self-paced learning mechanism is developed.