Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

@article{Li2021SemanticSW,
  title={Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization},
  author={Daiqing Li and Junlin Yang and Karsten Kreis and Antonio Torralba and Sanja Fidler},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={8296-8307}
}
  • Daiqing LiJunlin Yang S. Fidler
  • Published 12 April 2021
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts. This is the goal of semi-supervised learning, which exploits more widely available unlabeled data to complement small labeled data sets. In this paper, we propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. Concretely, we learn a generative adversarial network that captures the… 
[PDF] Semantic Reader
52 Citations
Highly Influential Citations
8
Background Citations
34
Methods Citations
25
Results Citations
4

Figures and Tables from this paper

52 Citations

Semi-Supervised Semantic Segmentation of Class-Imbalanced Images: A Hierarchical Self-Attention Generative Adversarial Network

  • Lu ChaiQinyuan Liu
  • Computer Science
    2022 7th International Conference on Image, Vision and Computing (ICIVC)
  • 2022
This work introduces a hierarchical generative model with a self-attention mechanism to help with capturing features of foreground objects and outperforms other baselines on semi-supervised segmentation of class-imbalanced images, and pre-serves out-of-domain generalization ability.

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

Through an extensive ablation study, this work shows big gains in leveraging a large generated dataset to train different supervised and self-supervised backbone models on pixel-wise tasks, and demonstrates that using the synthesized datasets for pre- training leads to improvements over standard ImageNet pre-training on several downstream datasets.

Learning to Annotate Part Segmentation with Gradient Matching

This paper focuses on tackling semi-supervised part segmentation tasks by generating high-quality images with a pre-trained GAN and labelling the generated images with an automatic annotator, and formulate the annotator learning as a learning-to-learn problem.

U-shaped GAN for Semi-Supervised Learning and Unsupervised Domain Adaptation in High Resolution Chest Radiograph Segmentation

This work improves GAN by replacing the traditional discriminator with a U-shaped net, which predicts each pixel a label, and extends it to UDA by taking the source and target domain data as the annotated data and the unannotated data in the semi-supervised learning approach, respectively.

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

This work introduces DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as the method.

Histopathology DatasetGAN: Synthesizing Large-Resolution Histopathology Datasets

This work proposes the Histopathology DatasetGAN (HDGAN) framework, an extension of the DatasetsetGAN framework for image generation and segmentation that scales well to large-resolution histopathology images.

Semi-supervised Semantic Segmentation with Error Localization Network

  • Donghyeon KwonSuha Kwak
  • Computer Science
    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2022
This paper presents a novel method that resolves the chronic issue of pseudo labeling in semi-supervised learning of semantic segmentation and introduces a new learning strategy for ELN that simulates plausible and diverse segmentation errors during training of ELN to enhance its generalization.

Dynamic-Pix2Pix: Noise Injected cGAN for Modeling Input and Target Domain Joint Distributions with Limited Training Data

This model surpasses the Pix2Pix model in segmenting HC18 and Montgomery’s chest x-ray images and produces comparable results for the in and out-domain generalization compared to the state-of-the-art methods.

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

This work introduces a generative adversarial network that can simultaneously generate aligned image samples from multiple related domains and proposes Polymorphic-GAN which learns shared features across all domains and a per-domain morph layer to morph shared features according to each domain.

One-Shot Synthesis of Images and Segmentation Masks

The OSMIS model is introduced, inspired by the recent architectural de-velopments of single-image GANs, which enables the synthesis of segmentation masks that are precisely aligned to the generated images in the one-shot regime, and outperforms state-of-the-art single- image GAN models in image synthesis quality and diversity.
...

References

SHOWING 1-10 OF 107 REFERENCES

Semi Supervised Semantic Segmentation Using Generative Adversarial Network

A semi-supervised framework is proposed – based on Generative Adversarial Networks (GANs) – which consists of a generator network to provide extra training examples to a multi-class classifier, acting as discriminator in the GAN framework, that assigns sample a label y from the K possible classes or marks it as a fake sample (extra class).

Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency

This work proposes an approach for semi-supervised semantic segmentation that learns from limited pixel-wise annotated samples while exploiting additional annotation-free images, and achieves significant improvement over existing methods, especially when trained with very few labeled samples.

Adversarial Learning for Semi-supervised Semantic Segmentation

It is shown that the proposed discriminator can be used to improve semantic segmentation accuracy by coupling the adversarial loss with the standard cross entropy loss of the proposed model.

A survey of semi- and weakly supervised semantic segmentation of images

This paper focuses on the core methods and reviews the semi- and weakly supervised semantic segmentation models in recent years, based on the commonly used models such as convolutional neural networks, fully Convolutional networks, generative adversarial networks.

Few-shot 3D Multi-modal Medical Image Segmentation using Generative Adversarial Learning

This work proposes a novel method based on Generative Adversarial Networks (GANs) to train a segmentation model with both labeled and unlabeled images, which prevents over-fitting by learning to discriminate between true and fake patches obtained by a generator network.

Unsupervised Data Augmentation for Consistency Training

A new perspective on how to effectively noise unlabeled examples is presented and it is argued that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.

Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning

An unsupervised loss function is proposed that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network.

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

This work introduces DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as the method.

Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing

This paper proposes to train a semantic segmentation network starting from the discriminative regions and progressively increase the pixel-level supervision using by seeded region growing, and obtains the state-of-the-art performance.

Big Self-Supervised Models are Strong Semi-Supervised Learners

The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2 (a modification of SimCLRs), supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.
...