On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation

@inproceedings{Jungo2018OnTE,
  title={On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation},
  author={Alain Jungo and Raphael Meier and Ekin Ermis and Marcela Blatti-Moreno and Evelyn Herrmann and Roland Wiest and Mauricio Reyes},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year={2018}
}
Uncertainty estimation methods are expected to improve the understanding and quality of computer-assisted methods used in medical applications (e.g., neurosurgical interventions, radiotherapy planning), where automated medical image segmentation is crucial. [] Key Result Additionally, we show that the learned observers' uncertainty can be combined with current standard Monte Carlo dropout Bayesian neural networks to characterize uncertainty of model's parameters.

Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle

A framework named Diagnosis First segmentation Framework (DiFF) is proposed to estimate diagnosis-first segmentation from the raw images, which will learn to fuse the multi-rater segmentation labels to a single ground-truth which could maximize the disease diagnosis performance.

Exploring the Relationship Between Segmentation Uncertainty, Segmentation Performance and Inter-observer Variability with Probabilistic Networks

This work shows how the area of disagreement between different ground-truth annotators can be developed into model confidence metrics and evaluated on the LIDC-IDRI dataset, which contains multiple expert annotations for each subject.

Uncertainty-Aware Training of Neural Networks for Selective Medical Image Segmentation

A novel method is presented that considers such uncertainty in the training process to maximize the accuracy on the confident subset rather than the Accuracy on the whole dataset.

Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations

Spatially Varying Label Smoothing (SVLS) is proposed, a soft labeling technique that captures the structural uncertainty in semantic segmentation and naturally lends itself to incorporate inter-rater uncertainty when multiple labelmaps are available.

Label fusion and training methods for reliable representation of inter-rater uncertainty

The results, across 10 data splittings on two public datasets, indicate that SoftSeg models, regardless of the ground truth fusion method, had better calibration and preservation of the inter-rater rater variability compared with their conventional counterparts without impacting the segmentation performance.

A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation

A systematic approach to quantitatively compare methods to compute epistemic uncertainty maps using the Monte-Carlo dropout technique to multi-class segmentation of the carotid artery lumen and vessel wall and made a python package available to reproduce the analysis on different data and tasks.

How Bad is Good enough: Noisy annotations for instrument pose estimation

This study shows the property of deep neural networks to learn dominant signals from noisy annotations with sufficient data in a regression setting by generating synthetic radiographs, where the ground truth pose is known and therefore the pose estimation error made by the medical-expert can be estimated from experiments.

Learning Calibrated Medical Image Segmentation via Multi-rater Agreement Modeling

  • Wei JiShuang Yu Yefeng Zheng
  • Computer Science
    2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2021
This work is the first in producing calibrated predictions under different expertise levels for medical image segmentation, and an expertise-aware inferring module or EIM is devised to embed the expertise level of individual raters as prior knowledge, to form high-level semantic features.

Uncertainty Quantification using Variational Inference for Biomedical Image Segmentation

  • Abhinav Sagar
  • Computer Science
    2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)
  • 2022
This work used an encoder decoder architecture based on variational inference techniques for segmenting brain tumour images using Dice Similarity Coefficient and Intersection Over Union as the evaluation metrics.

Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis

An overview of the existing methods to quantify uncertainty associated to DL predictions is proposed, which focuses on applications to medical image analysis, which present specific challenges due to the high dimensionality of images and their quality variability, as well as constraints associated to real-life clinical routine.
...

References

SHOWING 1-10 OF 11 REFERENCES

Validation of Image Segmentation and Expert Quality with an Expectation-Maximization Algorithm

An Expectation-Maximization algorithm for computing a probabilistic estimate of the "ground truth" segmentation from a group of expert segmentations, and a simultaneous measure of the quality of each expert is presented.

Clinical Evaluation of a Fully-automatic Segmentation Method for Longitudinal Brain Tumor Volumetry

Investigation of a fully-automatic segmentation method for longitudinal brain tumor volumetry by comparing the automatically estimated volumes with ground truth data acquired via manual segmentation suggests the potential of automated longitudinal tumor segmentation to substitute manual volumetric follow-up of contrast-enhancing and non-enhanced T2-hyperintense tumor compartments.

U-Net: Convolutional Networks for Biomedical Image Segmentation

It is shown that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks.

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

A Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty is presented, which makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.

A Practical Bayesian Framework for Backpropagation Networks

  • D. Mackay
  • Computer Science
    Neural Computation
  • 1992
A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks that automatically embodies "Occam's razor," penalizing overflexible and overcomplex models.

Weight Uncertainty in Neural Networks

This work introduces a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop, and shows how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems.

Weight Uncertainty in Neural Network

This work introduces a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop, and shows how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems.

Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference

This work presents an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches, and approximate the model's intractable posterior with Bernoulli variational distributions.

Deep Learning in Medical Image Analysis

Adam: A Method for Stochastic Optimization

This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.