• Corpus ID: 204960694

Small-GAN: Speeding Up GAN Training Using Core-sets

@inproceedings{Sinha2020SmallGANSU,
  title={Small-GAN: Speeding Up GAN Training Using Core-sets},
  author={Samarth Sinha and Hang Zhang and Anirudh Goyal and Yoshua Bengio and H. Larochelle and Augustus Odena},
  booktitle={ICML},
  year={2020}
}
Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the… 

Tables from this paper

Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
TLDR
A simple modification to the Generative Adversarial Network (GAN) training algorithm that materially improves results with no increase in computational cost is introduced: when updating the generator parameters, the gradient contributions from the elements of the batch that the critic scores as `least realistic' are zeroed out.
Freeze Discriminator: A Simple Baseline for Fine-tuning GANs
TLDR
It is shown that simple fine-tuning of GANs with frozen lower layers of the discriminator performs surprisingly well, and a simple baseline, FreezeD, significantly outperforms previous techniques used in both unconditional and conditional GAns.
Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
TLDR
This work suggests a brand-new angle towards data-efficient GAN training: identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same set.
Top-K Training of GANs: Improving Generators by Making Critics Less Critical
TLDR
A simple modification to the Generative Adversarial Network (GAN) training algorithm is introduced that materially improves results with no increase in computational cost: when updating the generator parameters, it is shown that this `top-k update' procedure is a generally applicable improvement.
Instance Selection for GANs
TLDR
This work proposes a novel approach to improve sample quality: altering the training dataset via instance selection before model training has taken place, which redirects model capacity towards high-density regions, which ultimately improves sample fidelity.
Ultra-Data-Efficient GAN Training: Drawing A Lottery Ticket First, Then Training It Toughly
TLDR
This work decomposes the data-hungry GAN training into two sequential sub-problems: identifying the lottery ticket from the original GAN; then training the found sparse subnetwork with aggressive data and feature augmentations, effectively stabilizing training and improving convergence.
GAN Training Acceleration Using Fréchet Descriptor-Based Coreset
TLDR
A Coreset using Fréchet Descriptor Distances (FDD-Coreset) is proposed to accelerate the training of GAN for blob identification and achieves higher denoising performance and maintains approximate performance of blob identification compared with training on the entire dataset.
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis
TLDR
This paper proposes a light-weight GAN structure that gains superior quality on 1024 × 1024 resolution, and shows its superior performance compared to the state-of-the-art StyleGAN2, when data and computing budget are limited.
HIGH-FIDELITY FEW-SHOT IMAGE SYNTHESIS
TLDR
This paper proposes a light-weight GAN structure that gains superior quality on 1024 × 1024 resolution, and shows its superior performance compared to the state-of-the-art StyleGAN2, when data and computing budget are limited.
Improved Consistency Regularization for GANs
TLDR
This work shows that consistency regularization can introduce artifacts into the GAN samples and proposes several modifications to the consistencyRegularization procedure designed to improve its performance, and yields the best known FID scores on various GAN architectures.
...
...

References

SHOWING 1-10 OF 79 REFERENCES
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
TLDR
This paper empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization and enable training visual recognition models on internet-scale data with high efficiency.
Do GANs learn the distribution? Some Theory and Empirics
Improved Training of Wasserstein GANs
TLDR
This work proposes an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input, which performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning.
Consistency Regularization for Generative Adversarial Networks
TLDR
This work proposes a simple, effective training stabilizer based on the notion of consistency regularization, which improves state-of-the-art FID scores for conditional generation and achieves the best F ID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA.
Self-Attention Generative Adversarial Networks
TLDR
The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
TLDR
This work proposes to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop, and constructs a new image dataset, LSUN, which contains around one million labeled images for each of 10 scene categories and 20 object categories.
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
TLDR
This work investigates the cause for this generalization drop in the large-batch regime and presents numerical evidence that supports the view that large- batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization.
Large Scale GAN Training for High Fidelity Natural Image Synthesis
TLDR
It is found that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input.
Prescribed Generative Adversarial Networks
TLDR
The prescribed GAN (PresGAN) is developed and found they mitigate mode collapse and generate samples with high perceptual quality and reduce the gap in performance in terms of predictive log-likelihood between traditional GANs and variational autoencoders (VAEs).
Improving GANs Using Optimal Transport
TLDR
Optimal Transport GAN (OT-GAN), a variant of generative adversarial nets minimizing a new metric measuring the distance between the generator distribution and the data distribution, resulting in a highly discriminative distance function with unbiased mini-batch gradients is presented.
...
...