Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy

  title={Modality Attention and Sampling Enables Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy},
  author={Alvaro Gomariz and Tiziano Portenier and Patrick M. Helbling and Stephan Isringhausen and Ute Suessbier and C{\'e}sar Nombela-Arrieta and Orcun Goksel},
  journal={Nature machine intelligence},
  pages={799 - 811}
Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable… 

Figures and Tables from this paper

Probabilistic Spatial Analysis in Quantitative Microscopy with Uncertainty-Aware Cell Detection using Deep Bayesian Regression of Density Maps
A deep-learning based cell detection framework that can operate on large microscopy images and outputs desired probabilistic predictions by integrating Bayesian techniques for the regression of uncertainty-aware density maps and a Probabilistic spatial analysis with Monte-Carlo sampling is proposed.
Utilizing Uncertainty Estimation in Deep Learning Segmentation of Fluorescence Microscopy Images with Missing Markers
A method to estimate segmentation quality on unlabeled images by estimating both aleatoric and epistemic uncertainties of convolutional neural networks for image segmentation, and training a Random Forest model for the interpretation of uncertainty features via regression to their corresponding segmentation metrics is proposed.
Chronic viral infections persistently alter marrow stroma and impair hematopoietic stem cell fitness.
It is demonstrated that chronic infection with lymphocytic choriomeningitis virus leads to long-lasting decimation of the BM stromal network of mesenchymal CXCL12-abundant reticular cells, and combined antibody-mediated inhibition of type I and II IFN pathways completely preempts degeneration of CARc and protects HSCs from chronic dysfunction.


Marker Sampling and Excite: Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy
  • CodeOcean
  • 2021
Modality attention and sampling enables deep learning with heterogeneous marker combinations in fluorescence microscopy
  • Nature Machine Intelligence
  • 2021
'Squeeze & Excite' Guided Few-Shot Segmentation of Volumetric Images
A novel few-shot framework, for the segmentation of volumetric medical images with only a few annotated slices, that incorporates 'squeeze & excite' blocks and outperforms multiple baselines and existing approaches with respect to the segmentations accuracy by a significant margin.
Live-animal imaging of native hematopoietic stem and progenitor cells
A dual genetic strategy is described in mice that restricts reporter labelling to a subset of the most quiescent long-term HSCs (LT-HSCs) and that is compatible with current intravital imaging approaches in the calvarial bone marrow, enabling the direct visualization of HSC behaviours and dissection of heterogeneity in HSC niches.
Mask R-CNN
This work presents a conceptually simple, flexible, and general framework for object instance segmentation that outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners.
NuSeT: A deep learning tool for reliably separating and analyzing crowded cells
NuSeT addresses common challenges in nuclear segmentation such as variability in nuclear signal and shape, limited training sample size, and sample preparation artifacts, and consistently fares better in generating accurate segmentation masks and assigning boundaries for touching nuclei.
Revealing architectural order with quantitative label-free imaging and deep learning
It is shown that experimental defects in labeling the human tissue can be rescued with quantitative label-free imaging and neural network model, and it is anticipated that the proposed method will enable new studies of architectural order at spatial scales ranging from organelles to tissue.
Squeeze-and-Excitation Networks
This work proposes a novel architectural unit, which is term the “Squeeze-and-Excitation” (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels and shows that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets.
'Project & Excite' Modules for Segmentation of Volumetric Medical Scans
This paper proposes `Project & Excite' (PE) modules that base upon the ideas of SE and extend them to operating on 3D volumetric images, and demonstrates that PE modules can be easily integrated in 3D U-Net, boosting performance by 5% Dice points, while only increasing the model complexity by 2%.
CollaGAN: Collaborative GAN for Missing Image Data Imputation
CollaGAN convert the image imputation problem to a multi-domain images-to-image translation task so that a single generator and discriminator network can successfully estimate the missing data using the remaining clean data set.