Region-aware Adaptive Instance Normalization for Image Harmonization

@article{Ling2021RegionawareAI,
  title={Region-aware Adaptive Instance Normalization for Image Harmonization},
  author={Jun Ling and Han Xue and Li Song and Rong Xie and Xiao Gu},
  journal={2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021},
  pages={9357-9366}
}
  • Jun Ling, Han Xue, Xiao Gu
  • Published 1 June 2021
  • Computer Science
  • 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Image composition plays a common but important role in photo editing. To acquire photo-realistic composite images, one must adjust the appearance and visual style of the foreground to be compatible with the background. Existing deep learning methods for harmonizing composite images directly learn an image mapping network from the composite to real one, without explicit exploration on visual style consistency between the background and the foreground images. To ensure the visual style… 

Figures and Tables from this paper

SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization
TLDR
A self-consistent style contrastive learning scheme (SCS-Co), which can learn more distortion knowledge and well regularize the generated harmonized image in the style representation space from two aspects of the foreground self-style and foreground-background style consistency, leading to a more photorealistic visual result.
NeurSF: Neural Shading Field for Image Harmonization
TLDR
This paper decomposes the image harmonization into two sub-problems: illumination estimation of background images and rendering of foreground objects and learns a direction-aware illumination descriptor via a neural rendering framework, of which the key is a Shading Module that decomposing the shading field into multiple shading components given depth information.
Making Images Real Again: A Comprehensive Survey on Deep Image Composition
TLDR
In this survey, the datasets and methods for the above research directions are summarized and the limitations and potential directions to facilitate the future research for image composition are discussed.
High-Resolution Image Harmonization via Collaborative Dual Transformations
TLDR
This work proposes a high-resolution image harmonization network with Collaborative Dual Transformation (CDTNet) to combine pixel- to-pixel transformation and RGB-to-RGB transformation coherently in an end-to -end network.
Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization
TLDR
A novel spatial-separated curve rendering network (SCRNet) for efficient and high-resolution image harmonization for the first time and achieves the state-of-the-art performance on both synthesized iHarmony4 and real-world DIH test set.
Image Harmonization by Matching Regional References
TLDR
This work firstly match the contents between the foreground and background and then adaptively adjust every foreground location according to the appearance of its content-related background regions, and designs a residual reconstruction strategy, that uses the predicted residual to adjust the appearance.
DRAN: Detailed Region-Adaptive Normalization for Conditional Image Synthesis
TLDR
Quantitative and qualitative experiments show that equipped with DRAN, the baseline models are able to achieve significant improvement in complex style transfer and texture details reconstruction.
Image Generation with Self Pixel-wise Normalization
TLDR
A novel normalization method, called self pixel-wise normalization (SPN), which effectively boosts the generative performance by performing the pixel-adaptive affine transformation without the mask image and is easily applicable in the existing generative models.
HYouTube: Video Harmonization Dataset
TLDR
A new video harmonization dataset HYouTube is constructed by adjusting the foreground of real videos to create synthetic composite videos and additionally create 100 real composite videos via copy-and-paste.
LayoutBERT: Masked Language Layout Model for Object Insertion
TLDR
This work proposes LayoutBERT, a novel self-supervised masked language model objective and bidirectional multi-head self-attention that outperforms previous layout-based likelihood models and shows favorable properties in terms of model capacity.
...
1
2
3
...

References

SHOWING 1-10 OF 45 REFERENCES
Deep Image Harmonization
TLDR
This work proposes an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization and introduces an efficient way to collect large-scale and high-quality training data that can facilitate the training process.
Improving the Harmony of the Composite Image by Spatial-Separated Attention Module
TLDR
This work proposes a novel attention module named Spatial-Separated Attention Module (S2AM) and designs a novel image harmonization framework by inserting the S2AM in the coarser low-level features of the Unet structure by two different ways.
Multi-scale image harmonization
TLDR
This work presents a framework that explicitly matches the visual appearance of images through a process the authors call image harmonization, before blending them, and shows how the proposed framework can be used to produce realistic composites with minimal user interaction in a number of different scenarios.
Image Style Transfer Using Convolutional Neural Networks
TLDR
A Neural Algorithm of Artistic Style is introduced that can separate and recombine the image content and style of natural images and provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation.
Semantic Image Synthesis With Spatially-Adaptive Normalization
TLDR
S spatially-adaptive normalization is proposed, a simple but effective layer for synthesizing photorealistic images given an input semantic layout that allows users to easily control the style and content of image synthesis results as well as create multi-modal results.
Learning Object Placement by Inpainting for Compositional Data Augmentation
TLDR
This work proposes a self-learning framework that automatically generates the necessary training data without any manual labeling by detecting, cutting, and inpainting objects from an image by proposing a PlaceNet that predicts a diverse distribution of common sense locations when given a foreground object and a background scene.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
TLDR
This work considers image transformation problems, and proposes the use of perceptual loss functions for training feed-forward networks for image transformation tasks, and shows results on image style transfer, where aFeed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time.
Style Transfer Via Texture Synthesis
TLDR
This paper proposes a novel style transfer algorithm that extends the texture synthesis work of Kwatra et al. (2005), while aiming to get stylized images that are closer in quality to the CNN ones.
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
TLDR
This paper presents a simple yet effective approach that for the first time enables arbitrary style transfer in real-time, comparable to the fastest existing approach, without the restriction to a pre-defined set of styles.
Few-Shot Unsupervised Image-to-Image Translation
TLDR
This model achieves this few-shot generation capability by coupling an adversarial training scheme with a novel network design, and verifies the effectiveness of the proposed framework through extensive experimental validation and comparisons to several baseline methods on benchmark datasets.
...
1
2
3
4
5
...