Unpaired Multi-Modal Segmentation via Knowledge Distillation
@article{Dou2020UnpairedMS, title={Unpaired Multi-Modal Segmentation via Knowledge Distillation}, author={Qi Dou and Quande Liu and Pheng-Ann Heng and Ben Glocker}, journal={IEEE Transactions on Medical Imaging}, year={2020}, volume={39}, pages={2415-2425} }
Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal…
Figures and Tables from this paper
64 Citations
Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation
- Computer ScienceMICCAI
- 2020
This paper designs class-specific affinity matrices to encode the knowledge of hierarchical feature reasoning, together with the shared convolutional layers to ensure the cross-modality generalization, and proposes an affinity-guided fully Convolutional network for multimodal image segmentation.
Multi-Task, Multi-Domain Deep Segmentation with Shared Representations and Contrastive Regularization for Sparse Pediatric Datasets
- Computer ScienceMICCAI
- 2021
This work proposes to train a segmentation model on multiple datasets, arising from different parts of the anatomy, in a multi-task and multi-domain learning framework, to overcome the inherent scarcity of pediatric data while benefiting from a more robust shared representation.
Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency
- Computer Science
- 2022
A novel scheme to achieve better pixel-level segmentation for unpaired multi-modal medical images by using a single Transformer with a carefully designed External Attention Module to learn the structured semantic consistency between modalities in the training phase.
MASS: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired CT and MRI images
- Computer ScienceMedical Image Analysis
- 2022
Domain Knowledge Driven Multi-modal Segmentation of Anatomical Brain Barriers to Cancer Spread
- Computer ScienceMICCAI
- 2020
A multi-modal segmentation method largely driven by domain knowledge is explored, which applies 3D U-Net as the backbone model and employs a label merging strategy for the symmetrical structures which have both left and right labels, to highlight the structural information regardless of the locations.
Structure-Driven Unsupervised Domain Adaptation for Cross-Modality Cardiac Segmentation
- Computer ScienceIEEE Transactions on Medical Imaging
- 2021
This paper presents a novel unsupervised domain adaptation framework for cross-modality cardiac segmentation, by explicitly capturing a common cardiac structure embedded across different modalities to guide cardiac segmentsation.
Incremental Learning Meets Transfer Learning: Application to Multi-site Prostate MRI Segmentation
- Computer ScienceArXiv
- 2022
This paper proposes a novel multi-site segmentation framework called incremental-transfer learning (ITL), which learns a model from multi- site datasets in an end-to-end sequential fashion, and shows for the first time that leveraging the ITL training scheme is able to alleviate challenging catastrophic forgetting problems in incremental learning.
Myocardial Pathology Segmentation of Multi-modal Cardiac MR Images with a Simple but Efficient Siamese U-shaped Network
- Computer ScienceBiomed. Signal Process. Control.
- 2022
Weakly supervised segmentation with cross-modality equivariant constraints
- Computer ScienceMedical Image Anal.
- 2022
Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to Unseen Domains
- Computer ScienceMICCAI
- 2020
A novel shape-aware meta-learning scheme is presented to improve the model generalization in prostate MRI segmentation, and outperforms many state-of-the-art generalization methods consistently across all six settings of unseen domains.
References
SHOWING 1-10 OF 41 REFERENCES
Multi-modal Learning from Unpaired Images: Application to Multi-organ Segmentation in CT and MRI
- Computer Science2018 IEEE Winter Conference on Applications of Computer Vision (WACV)
- 2018
Results demonstrate that information across modalities can in particular improve performance on varying structures such as the spleen, and show that multi-modal learning can improve overall accuracy over modality-specific training.
HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation
- Computer ScienceIEEE Transactions on Medical Imaging
- 2019
HyperDenseNet is proposed, a 3-D fully convolutional neural network that extends the definition of dense connectivity to multi-modal segmentation problems and has total freedom to learn more complex combinations between the modalities, within and in-between all the levels of abstraction, which increases significantly the learning representation.
SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth
- Computer ScienceIEEE Transactions on Medical Imaging
- 2019
An end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentationNetwork for a target imaging modality without having manual labels is proposed and achieved comparable performance to the traditional segmentationnetwork using target modality labels in certain scenarios.
Multi-label Whole Heart Segmentation Using CNNs and Anatomical Label Configurations
- Computer ScienceSTACOM@MICCAI
- 2017
Results on the MICCAI 2017 Multi-Modality Whole Heart Segmentation (MM-WHS) challenge show that the proposed architecture performs well on the provided CT and MRI training volumes, delivering in a three-fold cross validation an average Dice Similarity Coefficient over all heart substructures.
PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network at Unpaired Cross-Modality Cardiac Segmentation
- Computer ScienceIEEE Access
- 2019
A plug-and-play adversarial domain adaptation network (PnP-AdaNet) for adapting segmentation networks between different modalities of medical images, e.g., MRI and CT, which outperforms many state-of-the-art unsupervised domain adaptation approaches on the same dataset.
Learning Cross-Modality Representations From Multi-Modal Images
- Computer ScienceIEEE Transactions on Medical Imaging
- 2019
A shared autoencoder-like convolutional network that learns a common representation from multi-modal data is presented, and a form of feature normalization, a learning objective that minimizes cross-modality differences, and modality dropout are investigated, in which the network is trained with varying subsets of modalities.
VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images
- Computer ScienceNeuroImage
- 2018
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
- Computer Science2016 Fourth International Conference on 3D Vision (3DV)
- 2016
This work proposes an approach to 3D image segmentation based on a volumetric, fully convolutional, neural network, trained end-to-end on MRI volumes depicting prostate, and learns to predict segmentation for the whole volume at once.
Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network
- Computer Science2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- 2018
This work proposes a generic cross-modality synthesis approach and shows that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks.
Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
- Computer ScienceMedical Image Anal.
- 2017