ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing

@article{Lin2018STGANST,
  title={ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},
  author={Chen-Hsuan Lin and Ersin Yumer and Oliver Wang and Eli Shechtman and Simon Lucey},
  journal={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2018},
  pages={9455-9464}
}
We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image. To achieve this, we propose a novel Generative Adversarial Network (GAN) architecture that utilizes Spatial Transformer Networks (STNs) as the generator, which we call Spatial Transformer GANs (ST-GANs). ST-GANs seek image realism by operating in the geometric warp parameter space. In particular, we exploit an iterative STN warping scheme… Expand
Compositional GAN: Learning Conditional Image Composition
TLDR
This work proposes to model object composition in a GAN framework as a self-consistent composition-decomposition network and reveals that the learned model captures potential interactions between the two object domains given as input to output new instances of composed scene at test time in a reasonable fashion. Expand
Spatial Fusion GAN for Image Synthesis
TLDR
Qualitative and quantitative comparisons with the state-of-the-art demonstrate the superiority of the proposed Spatial Fusion GAN (SF-GAN), which combines a geometry synthesizer and an appearance synthesizer to achieve synthesis realism in both geometry and appearance spaces. Expand
Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks
TLDR
This work proposes an effective approach, termed as LoGAN, to support local editing of the output image of a well-trained GAN model and introduces two operators, i.e., content modulation and style modulation, together with a priority mask to facilitate the precise control of the intermediate generative features. Expand
Toward Realistic Image Compositing With Adversarial Learning
  • Bor-Chun Chen, Andrew Kae
  • Computer Science
  • 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2019
TLDR
Experimental results show that the proposed model, Geometrically and Color Consistent GANs (GCC-GANs), can automatically generate realistic composite images compared to several state-of-the-art methods, and does not require any manual effort. Expand
Towards Realistic 3D Embedding via View Alignment
TLDR
An innovative View Alignment GAN that composes new images by embedding 3D models into 2D background images realistically and automatically that achieves high-fidelity composition qualitatively and quantitatively as compared with state-of-the-art generation methods. Expand
Compositional GAN: Learning Image-Conditional Binary Composition
TLDR
This work proposes a novel self-consistent Composition-by-Decomposition network to compose a pair of objects and indicates that the learned model captures potential interactions between the two object domains, and generates realistic composed scenes at test time. Expand
Original Possion Blending Deep Harmonization Pix 2 Pix GCC-GAN ( Ours ) Foreground Mask
Compositing a realistic image is a challenging task and usually requires considerable human supervision using professional image editing software. In this work we propose a generative adversarialExpand
Vector Image Generation by Learning Parametric Layer Decomposition
Deep image generation is becoming a tool to enhance artists and designers creativity potential. In this paper, we aim at making the generation process easier to understand and interact with. InspiredExpand
A Layer-Based Sequential Framework for Scene Generation with GANs
TLDR
This work presents a scene generation framework based on Generative Adversarial Networks (GANs) to sequentially compose a scene, breaking down the underlying problem into smaller ones. Expand
Adversarial-learning-based image-to-image transformation: A survey
TLDR
This survey aims to provide an overview of adversarial-learning-based methods by focusing on the image-to-image transformation scenario and introduces the network architectures of generative models and loss functions. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 47 REFERENCES
LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
TLDR
The experiments demonstrate that LR-GAN can generate more natural images with objects that are more human recognizable than DCGAN and is trained in an end-to-end manner with gradient descent methods. Expand
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
We present a transformation-grounded image generation network for novel 3D view synthesis from a single image. Our approach first explicitly infers the parts of the geometry visible both in the inputExpand
Image-to-Image Translation with Conditional Adversarial Networks
TLDR
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Expand
BEGAN: Boundary Equilibrium Generative Adversarial Networks
TLDR
This work proposes a new equilibrium enforcing method paired with a loss derived from the Wasserstein distance for training auto-encoder based Generative Adversarial Networks, which provides a new approximate convergence measure, fast and stable training and high visual quality. Expand
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
TLDR
A novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets, and a powerful 3D shape descriptor which has wide applications in 3D object recognition. Expand
WarpNet: Weakly Supervised Matching for Single-View Reconstruction
TLDR
This work presents an approach to matching images of objects in fine-grained datasets without using part annotations, with an application to the challenging problem of weakly supervised single-view reconstruction, through a novel deep learning architecture, WarpNet, that aligns an object in one image with a different object in another. Expand
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
TLDR
This work presents an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples, and introduces a cycle consistency loss to push F(G(X)) ≈ X (and vice versa). Expand
Generative Visual Manipulation on the Natural Image Manifold
TLDR
This paper proposes to learn the natural image manifold directly from data using a generative adversarial neural network, and defines a class of image editing operations, and constrain their output to lie on that learned manifold at all times. Expand
Energy-based Generative Adversarial Network
We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold andExpand
Context Encoders: Feature Learning by Inpainting
TLDR
It is found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures, and can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods. Expand
...
1
2
3
4
5
...