• Corpus ID: 204509684

Implicit competitive regularization in GANs

@article{Schfer2020ImplicitCR,
  title={Implicit competitive regularization in GANs},
  author={Florian Sch{\"a}fer and Hongkai Zheng and Anima Anandkumar},
  journal={ArXiv},
  year={2020},
  volume={abs/1910.05852}
}
To improve the stability of GAN training we need to understand why they can produce realistic samples. Presently, this is attributed to properties of the divergence obtained under an optimal discriminator. This argument has a fundamental flaw: If we do not impose regularity of the discriminator, it can exploit visually imperceptible errors of the generator to always achieve the maximal generator loss. In practice, gradient penalties are used to regularize the discriminator. However, this needs… 

Figures from this paper

Stabilizing Generative Adversarial Networks: A Survey

A comprehensive overview of the GAN training stabilization methods is provided, which discusses the advantages and disadvantages of each approach, offers a comparative summary, and concludes with a discussion of open problems.

Do GANs always have Nash equilibria?

It is proved that the optimal generative model in Wasserstein GAN problems provides a proximal equilibrium, which captures the sequential nature of GANs, in which the generator moves first followed by the discriminator.

Train simultaneously, generalize better: Stability of gradient-based minimax learners

The success of minimax learning problems of generative adversarial networks (GANs) has been observed to depend on the minimax optimization algorithm used for their training, and it is shown that the optimization algorithm also plays a key role in the generalization performance of the trained minimax model.

Generative Minimization Networks: Training GANs Without Competition

This work proposes to address the problem of min-max structure of the GAN objective by optimizing a different objective that circumvents the min- max structure using the notion of duality gap from game theory and provides novel convergence guarantees on this objective.

Approximation and convergence of GANs training: an SDE approach

This paper establishes approximations, with precise error bound analysis, for the training of GANs under stochastic gradient algorithms (SGAs) in the form of coupled Stochastic differential equations (SDEs).

Complex Momentum for Learning in Games

It is empirically demonstrate that complex-valued momentum can improve convergence in adversarial games—like generative adversarial networks—by showing it can find better solutions with an almost identical computational cost.

A case for new neural network smoothness constraints

This work tackles the question of model smoothness and shows that it is a useful inductive bias which aids generalization, adversarial robustness, generative modeling and reinforcement learning and concludes that new advances in the field are hinging on finding ways to incorporate data, tasks and learning into the definitions of smoothness.

A Limited-Capacity Minimax Theorem for Non-Convex Games or: How I Learned to Stop Worrying about Mixed-Nash and Love Neural Nets

The main contribution is to provide an approximate minimax theorem for a large class of games where the players pick neural networks including WGAN, StarCraft II and Blotto Game, which relies on the fact that despite being nonconcave-nonconvex with respect to the neural networks parameters, these games are concave-conveX with respectto the actual models represented by these neural networks.

Competitive Mirror Descent

This work proposes competitive mirror descent (CMD): a general method for solving constrained competitive optimization problems based on first order information that can be obtained by automatic differentiation, including a novel competitive multiplicative weights algorithm for problems on the positive cone.

Graph Generative Adversarial Networks for Sparse Data Generation in High Energy Physics

A graph generative adversarial network is developed to generate sparse data sets like those produced at the CERN Large Hadron Collider, and the model successfully generates sparse MNIST digits and particle jet data.

References

SHOWING 1-10 OF 62 REFERENCES

Improved Training of Wasserstein GANs

This work proposes an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input, which performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning.

Stabilizing Training of Generative Adversarial Networks through Regularization

This work proposes a new regularization approach with low computational cost that yields a stable GAN training procedure and demonstrates the effectiveness of this regularizer accross several architectures trained on common benchmark image generation tasks.

Generalization and equilibrium in generative adversarial nets (GANs) (invited talk)

Generative Adversarial Networks (GANs) have become one of the dominant methods for fitting generative models to complicated real-life data, and even found unusual uses such as designing good

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

This work proposes a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions and introduces the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score.

Are GANs Created Equal? A Large-Scale Study

A neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures finds that most models can reach similar scores with enough hyperparameter optimization and random restarts, suggesting that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes.

On Convergence and Stability of GANs

This work proposes studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions, and shows that DRAGAN enables faster training, achieves improved stability with fewer mode collapses, and leads to generator networks with better modeling performance across a variety of architectures and objective functions.

Training GANs with Optimism

This work addresses the issue of limit cycling behavior in training Generative Adversarial Networks and proposes the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs and introduces a new algorithm, Optimistic Adam, which is an optimistic variant of Adam.

Adversarial Divergences are Good Task Losses for Generative Modeling

It is argued that adversarial learning, pioneered with generative adversarial networks (GANs), provides an interesting framework to implicitly define more meaningful task losses for generative modeling tasks, such as for generating "visually realistic" images.

A Closer Look at the Optimization Landscapes of Generative Adversarial Networks

New visualization techniques for the optimization landscapes of GANs are proposed that enable us to study the game vector field resulting from the concatenation of the gradient of both players.

A Style-Based Generator Architecture for Generative Adversarial Networks

An alternative generator architecture for generative adversarial networks is proposed, borrowing from style transfer literature, that improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation.
...