Repurposing GANs for One-shot Semantic Part Segmentation

@article{Tritrong2021RepurposingGF,
  title={Repurposing GANs for One-shot Semantic Part Segmentation},
  author={Nontawat Tritrong and Pitchaporn Rewatbowornwong and Supasorn Suwajanakorn},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={4473-4483}
}
While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? In this work, we test this hypothesis and propose a simple and effective approach based on GANs for semantic part segmentation that requires as few as one label example along with an unlabeled dataset. Our key idea is to leverage a trained GAN to extract a… 

Figures and Tables from this paper

Dual-Domain Image Synthesis using Segmentation-Guided GAN
TLDR
A segmentation-guided approach to synthesise images that integrate features from two distinct domains, which combines few-shot cross-domain StyleGAN with a latent optimiser to achieve images containing features of two separate domains.
GANORCON: Are Generative Models Useful for Few-shot Segmentation?
TLDR
These experiments suggest that the inductive biases of generative models, such as their ability to dis-entangle shape and texture, are well captured by standard feed-forward networks trained using contrastive learning.
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
TLDR
By training an effective feature segmentation architecture on top of BigGAN, it is shown that VQGAN can similarly serve as a dataset generator, leveraging the already annotated data, and creates a new ImageNet benchmark by labeling an additional set of real images and evaluating segmentation performance in a variety of settings.
Interpreting Class Conditional GANs with Channel Awareness
TLDR
Channel awareness is proposed, which quantitatively characterizes how a single channel contributes to the final synthesis of multiple classes in a class conditional generator, and achieves versatile image editing via simply altering a single channels and manages to harmoniously hybridize two different classes.
State-of-the-Art in the Architecture, Methods and Applications of StyleGAN
TLDR
This state-of-the-art report covers the StyleGAN architecture, and the ways it has been employed since its conception, while also analyzing its severe limitations.
CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions
TLDR
This work investigates how to effectively link the pretrained latent spaces of StyleGAN and CLIP, which in turn allows us to automatically extract semantically labeled edit directions from StyleGAN, finding and naming meaningful edit operations without any additional human guidance.
Ensembling with Deep Generative Views
TLDR
This work uses StyleGAN2 as the source of generative augmentations and investigates whether such views can be applied to real images to benefit downstream analysis tasks such as image classification.
GAN Inversion: A Survey
TLDR
This paper provides a survey of GAN inversion with a focus on its representative algorithms and its applications in image restoration and image manipulation, and discusses the trends and challenges for future research.
GAN-Supervised Dense Visual Alignment
TLDR
GANgealing significantly outperforms past self-supervised correspondence algorithms and performs on-par with state-of-the-art supervised correspondence algorithms on several datasets—without making use of any correspondence supervision or data augmentation and despite being trained exclusively on GAN-generated data.
Generative Models as a Data Source for Multiview Representation Learning
TLDR
This paper compares several representation learning methods that can be applied to the setting of learning general-purpose visual representations from a black-box generative model rather than directly from data, and finds that the resulting representations rival those learned directly from real data.
...
1
2
...

References

SHOWING 1-10 OF 65 REFERENCES
Analyzing and Improving the Image Quality of StyleGAN
TLDR
This work redesigns the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images, and thereby redefines the state of the art in unconditional image modeling.
Deep Learning for Semantic Part Segmentation with High-Level Guidance
TLDR
A state-of-the-art semantic segmentation system is adapted to this task, and it is shown that a combination of a fully-convolutional Deep CNN system coupled with Dense CRF labelling provides excellent results for a broad range of object categories.
Unet: Convolutional networks for biomedical image segmentation
  • In International Conference on Medical image computing and computer-assisted intervention,
  • 2015
Pose-Guided Knowledge Transfer for Object Part Segmentation
TLDR
This paper proposes and evaluates a new model for transferring part segmentations using keypoints, even for complex object poses and across different object classes.
Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer
TLDR
A novel method to generate synthetic human part segmentation data using easily-obtained human keypoint annotations to exploit the anatomical similarity among human to transfer the parsing results of a person to another person with similar pose.
Motion-supervised Co-Part Segmentation
TLDR
This work proposes a self-supervised deep learning method for co-part segmentation that develops the idea that motion information inferred from videos can be leveraged to discover meaningful object parts.
Ordinal Multi-Task Part Segmentation With Recurrent Prior Generation
TLDR
This work proposes a novel Ordinal Multi-task Part Segmentation (OMPS) approach which explicitly models the part ordinal relationship to guide the segmentation process in a recurrent manner and outperforms the state-of-the-art models on human and vehicle part parsing benchmarks.
Correlating Edge, Pose With Parsing
TLDR
This paper proposes a CorrPM employing a heterogeneous non-local block to discover the spatial affinity among feature maps from the edge, pose and parsing and finds that uncovering the correlation among the three factors is a superior way of leveraging the pivotal contextual cues provided by edges and poses.
Editing in Style: Uncovering the Local Semantics of GANs
TLDR
A simple and effective method for making local, semantically-aware edits to a target output image via a novel manipulation of style vectors that relies on the emergent disentanglement of semantic objects learned by StyleGAN during its training.
MaskGAN: Towards Diverse and Interactive Facial Image Manipulation
TLDR
This work proposes a novel framework termed MaskGAN, enabling diverse and interactive face manipulation, and finds that semantic masks serve as a suitable intermediate representation for flexible face manipulation with fidelity preservation.
...
1
2
3
4
5
...