Mitigating Uncertainty in Document Classification

@inproceedings{Zhang2019MitigatingUI,
  title={Mitigating Uncertainty in Document Classification},
  author={Xuchao Zhang and Fanglan Chen and Chang-Tien Lu and Naren Ramakrishnan},
  booktitle={North American Chapter of the Association for Computational Linguistics},
  year={2019}
}
The uncertainty measurement of classifiers' predictions is especially important in applications such as medical diagnoses that need to ensure limited human resources can focus on the most uncertain predictions returned by machine learning models. [] Key Method We also design a metric learning method on feature representations, which can boost the performance of dropout-based uncertainty methods with smaller prediction variance in accurate prediction trials. Extensive experiments on real-world data sets…
[PDF] Semantic Reader
22 Citations
Highly Influential Citations
3
Background Citations
14
Methods Citations
10

Figures and Tables from this paper

22 Citations

Towards More Accurate Uncertainty Estimation in Text Classification

A model called MSD is proposed to improve the accuracy of uncertainty score by reducing the effect of overconfidence of winning score and considering the impact of different categories of uncertainty simultaneously, which can be applied with different Deep Neural Networks.

Uncertainty-Aware Reliable Text Classification

This paper proposes an inexpensive framework that adopts both auxiliary outliers and pseudo off-manifold samples to train the model with prior knowledge of a certain class, which has high vacuity for OOD samples, and demonstrates that the model based on evidential uncertainty outperforms other counterparts for detecting OOD examples.

Revisiting Softmax for Uncertainty Approximation in Text Classification

A thorough empirical analysis of Monte Carlo Dropout and softmax is performed, suggesting that softmax can in fact be a sufficient uncertainty estimate when putational resources are a concern.

Uncertainty Estimation of Transformer Predictions for Misclassification Detection

A vast empirical investigation of state-of-the-art UE methods for Transformer models on misclassification detection in named entity recognition and text classification tasks and two computationally efficient modifications are proposed, one of which approaches or even outperforms computationally intensive methods.

A Concept-based Abstraction-Aggregation Deep Neural Network for Interpretable Document Classification

A corpus-level explanation approach is proposed, which aims to capture causal relationships between keywords and model predictions via learning importance of keywords for predicted labels across a training corpus based on attention weights.

Corpus-level and Concept-based Explanations for Interpretable Document Classification

A corpus-level explanation approach, which aims at capturing causal relationships between keywords and model predictions via learning the importance of keywords for predicted labels across a training corpus based on attention weights, and a concept-based explanation method that can automatically learn higher level concepts and their importance to model prediction tasks.

A Bayesian end-to-end model with estimated uncertainties for simple question answering over knowledge bases

Entropy methods for the confidence assessment of probabilistic classification models

It is argued that part of the information that is discarded in this process can be in fact used to further evaluate the goodness of models, and in particular the confidence with which each prediction is made.

An Interpretable and Uncertainty Aware Multi-Task Framework for Multi-Aspect Sentiment Analysis

An Attention-driven Keywords Ranking (AKR) method, which can automatically discover aspect keywords and aspect-level opinion keywords from the review corpus based on the attention weights, is proposed, which is significant for rating predictions by FEDAR.

Deliberate Self-Attention Network with Uncertainty Estimation for Multi-Aspect Review Rating Prediction

An Attention-driven Keywords Ranking (AKR) method, which can automatically extract aspect-level sentiment-related keywords from the review corpus based on the attention weights, and a LEcture-AuDience strategy to estimate model uncertainty in the context of multi-task learning, so that valuable human resources can focus on the most uncertain predictions.

References

SHOWING 1-10 OF 33 REFERENCES

Distance-based Confidence Score for Neural Network Classifiers

This paper proposes a simple, scalable method to achieve a reliable confidence score, based on the data embedding derived from the penultimate layer of the network, and investigates two ways to achieve desirable embeddings.

Confidence Measures for Neural Network Classifiers

A simple measure is proposed which combines the benefits of the probabi-listic interpretation of network outputs and the estimation of the quality of the model by bootstrap error estimation and behaves often better than more sophisticated measures, but may be dangerous under certain situations.

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

This work proposes an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates.

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

A new theoretical framework is developed casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes, which mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy.

From sBoW to dCoT marginalized encoders for text representation

This paper proposes Dense Cohort of Terms (dCoT), an unsupervised algorithm to learn improved sBoW document features and demonstrates empirically, on several benchmark datasets, that dCoT features significantly improve the classification accuracy across several document classification tasks.

NewsWeeder: Learning to Filter Netnews

A tutorial on conformal prediction

This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples of how the model under which successive examples are sampled independently from the same distribution can be applied to any method for producing ŷ.

Bayesian LSTMs in medicine

The utility of using Bayesian LSTMs for classification of medical time series is demonstrated and cherry-picked examples of confident and uncertain classifications of themedical time series are shown.

Explaining and Harnessing Adversarial Examples

It is argued that the primary cause of neural networks' vulnerability to adversarial perturbation is their linear nature, supported by new quantitative results while giving the first explanation of the most intriguing fact about them: their generalization across architectures and training sets.

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

A Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty is presented, which makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.