Semantic Palette: Guiding Scene Generation with Class Proportions

  title={Semantic Palette: Guiding Scene Generation with Class Proportions},
  author={Guillaume Le Moing and Tuan-Hung Vu and Himalaya Jain and Patrick P'erez and Matthieu Cord},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  • G. L. Moing, Tuan-Hung Vu, +2 authors M. Cord
  • Published 1 June 2021
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Despite the recent progress of generative adversarial networks (GANs) at synthesizing photo-realistic images, producing complex urban scenes remains a challenging problem. Previous works break down scene generation into two consecutive phases: unconditional semantic layout synthesis and image synthesis conditioned on layouts. In this work, we propose to condition layout generation as well for higher semantic control: given a vector of class proportions, we generate layouts with matching… 
1 Citations
SAC-GAN: Structure-Aware Image-to-Image Composition for Self-Driving
This work presents an end-to-end neural network trained to seamlessly compose an object represented as a cropped patch from an object image, into a background scene image, and evaluates the network in terms of quality, composability, and generalizability of the composite images.


Semantic Bottleneck Scene Generation
This work proposes a semantic bottleneck GAN model for unconditional synthesis of complex scenes that outperforms state-of-the-art generative models in unsupervised image synthesis on two challenging domains in terms of the Frechet Inception Distance and user-study evaluations.
Decomposing Image Generation into Layout Prediction and Conditional Synthesis
This article investigates splitting the optimisation of generative adversarial networks into two parts, by first generating a semantic segmentation masks from noise and then translating that segmentation mask into an image.
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
A new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs) is presented, which significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.
Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
This work argues that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images, and proposes a feature pyramid semantics-embedding discriminator, which is more effective in enhancing fine details and semantic alignments between the generated images and the input semantic layouts.
Analyzing and Improving the Image Quality of StyleGAN
This work redesigns the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images, and thereby redefines the state of the art in unconditional image modeling.
Seeing What a GAN Cannot Generate
This work visualize mode collapse at both the distribution level and the instance level, and deploys a semantic segmentation network to compare the distribution of segmented objects in the generated images with the target distribution in the training set.
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
This work proposes two novel, complementary methods using (i) entropy loss and (ii) adversarial loss respectively for unsupervised domain adaptation in semantic segmentation with losses based on the entropy of the pixel-wise predictions.
Semantic Image Synthesis With Spatially-Adaptive Normalization
S spatially-adaptive normalization is proposed, a simple but effective layer for synthesizing photorealistic images given an input semantic layout that allows users to easily control the style and content of image synthesis results as well as create multi-modal results.
A Style-Based Generator Architecture for Generative Adversarial Networks
  • Tero Karras, S. Laine, Timo Aila
  • Computer Science, Mathematics
    2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
An alternative generator architecture for generative adversarial networks is proposed, borrowing from style transfer literature, that improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation.
Image-to-Image Translation with Conditional Adversarial Networks
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.