Sketch Your Own GAN

  title={Sketch Your Own GAN},
  author={Sheng-Yu Wang and David Bau and Junyan Zhu},
  journal={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
Can a user create a deep generative model by sketching a single example? Traditionally, creating a GAN model has required the collection of a large-scale dataset of exemplars and specialized knowledge in deep learning. In contrast, sketching is possibly the most universally accessible way to convey a visual concept. In this work, we present a method, GAN Sketching, for rewriting GANs with one or more sketches, to make GANs training easier for novice users. In particular, we change the weights… 

Rewriting geometric rules of a GAN

This work enables a user to "warp" a given model by editing just a handful of original model outputs with desired geometric changes, enabling the creation of a new generative model without the burden of curating a large-scale dataset.

Diversifying detail and appearance in sketch-based face image synthesis

Quantitative and quantitative evaluations demonstrate that the synthesis of diverse face images from a single sketch by using a three-stage framework consisting of sketch refinement, detail enhancement, and appearance synthesis offers greater diversity in its output images than the state-of-the-art methods, while retaining the output realism.

Stabilizing Generative Models using Self-Supervision

This work introduces a novel self-supervised loss that encourages the discriminator to approximate a richer reward function, and employs the method to train a graph-based multi-agent actor-critic architecture that learns a centralized policy, conditioned on a learned latent interaction graph.

Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

A stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval, and can be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.

SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches

This work investigates a new paradigm of sketch-based image manipulation: mask-free local image manipulation, which only requires sketch inputs from users and utilizes the entire original image.

State‐of‐the‐Art in the Architecture, Methods and Applications of StyleGAN

This state‐of‐the‐art report covers the StyleGAN architecture, and the ways it has been employed since its conception, while also analyzing its severe limitations.

Controlling StyleGANs using rough scribbles via one‐shot learning

A simple mapping between StyleGAN features and each semantic class from a single example of semantic scribbles is constructed, which can generate an unlimited number of pseudo semantic scribble from random noise to train an encoder for controlling a pre-trained StyleGAN generator.

Sketch-based Facial Synthesis: A New Challenge

A high-quality dataset for FSS is introduced, named FS2K, which consists of 2,104 image-sketch pairs spanning three types of sketch styles, image backgrounds, lighting conditions, skin colors, and facial attributes and surpasses the performance of all previous state-of-the-art models on the proposedFS2K dataset by a large margin.

Facial-sketch Synthesis: A New Challenge

This paper introduces a high-quality dataset for FSS, named FS2K, which consists of 2 104 image-sketch pairs spanning three types of sketch styles, image backgrounds, lighting conditions, skin colors, and facial attributes, and presents the largest-scale FSS investigation by reviewing 89 classic methods.

Deep Facial Synthesis: A New Challenge

This paper introduces a high-quality dataset for FSS, named FS2K, which consists of 2,104 image-sketch pairs spanning three types of sketch styles, image backgrounds, lighting conditions, skin colors, and facial attributes, and presents the largest-scale FSS study by investigating 139 classical methods.



Scribbler: Controlling Deep Image Synthesis with Sketch and Color

A deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces is proposed and demonstrates a sketch based image synthesis system which allows users to scribble over the sketch to indicate preferred color for objects.

SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

  • W. ChenJames Hays
  • Computer Science
    2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • 2018
This work proposes a novel Generative Adversarial Network approach that synthesizes plausible images from 50 categories including motorcycles, horses and couches and introduces a new network building block suitable for both the generator and discriminator which improves the information flow by injecting the input image at multiple scales.

In-Domain GAN Inversion for Real Image Editing

An in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing, which achieves satisfying real image reconstruction and facilitates various image editing tasks, significantly outperforming start-of-the-arts.

Sketch-a-Net: A Deep Neural Network that Beats Humans

It is shown that state-of-the-art deep networks specifically engineered for photos of natural objects fail to perform well on sketch recognition, regardless whether they are trained using photos or sketches.

Large Scale GAN Training for High Fidelity Natural Image Synthesis

It is found that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input.

Generative Visual Manipulation on the Natural Image Manifold

This paper proposes to learn the natural image manifold directly from data using a generative adversarial neural network, and defines a class of image editing operations, and constrain their output to lie on that learned manifold at all times.

A Style-Based Generator Architecture for Generative Adversarial Networks

An alternative generator architecture for generative adversarial networks is proposed, borrowing from style transfer literature, that improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation.

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

A new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs) is presented, which significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.

Improved Techniques for Training GANs

This work focuses on two applications of GANs: semi-supervised learning, and the generation of images that humans find visually realistic, and presents ImageNet samples with unprecedented resolution and shows that the methods enable the model to learn recognizable features of ImageNet classes.

Differentiable Augmentation for Data-Efficient GAN Training

DiffAugment is a simple method that improves the data efficiency of GANs by imposing various types of differentiable augmentations on both real and fake samples, and can generate high-fidelity images using only 100 images without pre-training, while being on par with existing transfer learning algorithms.