RS-Net: Regression-Segmentation 3D CNN for Synthesis of Full Resolution Missing Brain MRI in the Presence of Tumours

@article{Mehta2018RSNetR3,
  title={RS-Net: Regression-Segmentation 3D CNN for Synthesis of Full Resolution Missing Brain MRI in the Presence of Tumours},
  author={Raghav Mehta and Tal Arbel},
  journal={ArXiv},
  year={2018},
  volume={abs/1807.10972}
}
Accurate synthesis of a full 3D MR image containing tumours from available MRI (e.g. to replace an image that is currently unavailable or corrupted) would provide a clinician as well as downstream inference methods with important complementary information for disease analysis. [] Key Result The system further provides uncertainty estimates based on Monte Carlo (MC) dropout [11] for the synthesized volume at each voxel, permitting quantification of the system’s confidence in the output at each location.
Medical Image Segmentation on MRI Images with Missing Modalities: A Review
TLDR
The main goal of this research is to offer a performance evaluation of missing modality compensating networks, as well as to outline the future strategies for dealing with this issue.
HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast Images
TLDR
HAD-Net is presented, a novel offline adversarial knowledge distillation (KD) technique, whereby a pre-trained teacher segmentation network, with access to all MRI sequences, teaches a student network, via hierarchical adversarial training, to better overcome the large domain shift presented when crucial images are absent during inference.
DropFilter for Uncertainty Estimation in Brain Tumor Segmentation
TLDR
The models are able to efficiently and harmlessly incorporate the idea of Dropout for convolutional neural network architectures, which is known as Spatial Dropout, or DropFilter, into Bayesian inference based on the MC sampling.
Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network
TLDR
A variant of generative adversarial network capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan is proposed and compared with competing unimodal and multi-modal methods.
Prior-Guided Image Reconstruction for Accelerated Multi-Contrast MRI via Generative Adversarial Networks
TLDR
A new approach for synergistic recovery of undersampled multi-contrast acquisitions based on conditional generative adversarial networks is proposed, which mitigates the limitations of pure learning-based reconstruction or synthesis by utilizing three priors: shared high-frequency prior available in the source contrast to preserve high-spatial-frequency details, low-frequencyPrior available inThe undersampling target contrast to prevent feature leakage/loss, and perceptual prior to improve recovery of high-level features.
mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis
Multimodal MRI Synthesis Using Unified Generative Adversarial Networks.
TLDR
The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image, and shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input.
PVT: Point-Voxel Transformer for 3D Deep Learning
In this paper, we present an efficient and high-performance neural architecture, termed Point-Voxel Transformer (PVT) for 3D deep learning, which deeply integrates both 3D voxelbased and point-based
Point-Voxel Transformer: An Efficient Approach To 3D Deep Learning
TLDR
This work presents a novel 3D Transformer, called Point-Voxel Transformer (PVT) that leverages self-attention computation in points to gather global context features, while performing multi-head self-ATTention (MSA) computation in voxels to capture local information and reduce the irregular data access.
Deep learning for brain disorders: from data processing to disease treatment
TLDR
The use of deep learning for brain disorders is reviewed, including the main applications, the concerned disorders and the types of architectures and data used, and guidelines to bridge the gap between research studies and clinical routine are provided.
...
...

References

SHOWING 1-10 OF 18 REFERENCES
HeMIS: Hetero-Modal Image Segmentation
TLDR
A deep learning image segmentation framework that is extremely robust to missing imaging modalities, which learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations are well defined.
Why Does Synthesized Data Improve Multi-sequence Classification?
TLDR
Experiments with two classifiers, linear support vector machines SVMs and random forests, together with two synthesis methods that can replace missing data in an image classification problem: neural networks and restricted Boltzmann machines RBMs are presented.
Deep MR to CT Synthesis Using Unpaired Data
TLDR
This work proposes to train a generative adversarial network (GAN) with unpaired MR and CT images to synthesize CT images that closely approximate reference CT images, and was able to outperform a GAN model trained with paired MR andCT images.
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
TLDR
The proposed network extends the previous u-net architecture from Ronneberger et al. by replacing all 2D operations with their 3D counterparts and performs on-the-fly elastic deformations for efficient data augmentation during training.
The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)
TLDR
The set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences are reported, finding that different algorithms worked best for different sub-regions, but that no single algorithm ranked in the top for all sub-Regions simultaneously.
Cross-Domain Synthesis of Medical Images Using Efficient Location-Sensitive Deep Network
TLDR
A novel architecture called location-sensitive deep network LSDN is proposed for synthesizing images across domains that integrates intensity feature from image voxels and spatial information in a principled manner and is computationally efficient, e.g. 26× faster than other sparse representation based methods.
A Compressed Sensing Approach for MR Tissue Contrast Synthesis
TLDR
Experiments on real data, obtained using different scanners and pulse sequences, show improvement in segmentation consistency, which could be extremely valuable in the pooling of multi-site multi-scanner neuroimaging studies.
Image-to-Image Translation with Conditional Adversarial Networks
TLDR
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.
Adam: A Method for Stochastic Optimization
TLDR
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
...
...