Coherent Online Video Style Transfer

@article{Chen2017CoherentOV,
  title={Coherent Online Video Style Transfer},
  author={Dongdong Chen and Jing Liao and Lu Yuan and Nenghai Yu and Gang Hua},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
  year={2017},
  pages={1114-1123}
}
  • Dongdong Chen, Jing Liao, G. Hua
  • Published 27 March 2017
  • Computer Science
  • 2017 IEEE International Conference on Computer Vision (ICCV)
Training a feed-forward network for the fast neural style transfer of images has proven successful, but the naive extension of processing videos frame by frame is prone to producing flickering results. We propose the first end-toend network for online video style transfer, which generates temporally coherent stylized video sequences in near realtime. Two key ideas include an efficient network by incorporating short-term coherence, and propagating short-term coherence to long-term, which ensures… 

Figures and Tables from this paper

ReCoNet: Real-time Coherent Video Style Transfer Network
TLDR
A novel real-time video style transfer model, ReCoNet, is proposed, which can generate temporally coherent style transfer videos while maintaining favorable perceptual styles and exhibits outstanding performance both qualitatively and quantitatively.
Real-time Coherent Video Style Transfer
TLDR
ReCoNet is a feed-forward convolutional neural network which can generate temporally coherent style transfer videos in real-time speed and adopts a novel luminance warping constraint to increase stylization stability under illumination effects and a novel feature-maplevel temporal loss to further enhance temporal consistency on traceable objects.
Real-time arbitrary video style transfer
TLDR
The experimental results show that the first real-time arbitrary video style transfer method with only one model outperforms the state-of-the-arts.
Real-time Coherent Video Style Transfer Final Report
  • Computer Science
  • 2018
TLDR
In ReCoNet, a novel luminance warping constraint is added to the temporal loss at the output level to capture luminance changes between consecutive frames and increase stylization stability under illumination effects and a new feature-maplevel temporal loss to further improve temporal consistency on the appearances of traceable objects.
Deliberation on object-aware video style transfer network with long-short temporal and depth-consistent constraints
TLDR
The main effort of this work is to propose an efficient salient object-aware and depth-consistent video style transfer algorithm, and DenseNet is carefully extended as feed-forward backbone network for better style transfer quality.
Arbitrary Video Style Transfer via Multi-Channel Correlation
TLDR
Multi-Channel Correction network (MCCNet) is proposed, which can be trained to fuse the exemplar style features and input content features for efficient style transfer while naturally maintaining the coherence of input videos.
Fast Video Multi-Style Transfer
TLDR
This work designs a multi-instance normalization block (MIN-Block) to learn different style examples and two ConvLSTM modules to encourage the temporal consistency and demonstrates to be able to generate temporally-consistent video transfer results in different styles while keeping each stylized frame visually pleasing.
Consistent Video Style Transfer via Relaxation and Regularization
TLDR
This article identifies the cause of the conflict between style transfer and temporal consistency, and proposes to reconcile this contradiction by relaxing the objective function, so as to make the stylization loss term more robust to motions.
Consistent Video Style Transfer via Compound Regularization
TLDR
Experimental results show that the proposed regularization can better balance the spatial and temporal performance, which supports the modeling and demonstrates the superiority of the method over other state-of-the-art style transfer methods.
Learning Blind Video Temporal Consistency
TLDR
An efficient approach based on a deep recurrent network for enforcing temporal consistency in a video that can handle multiple and unseen tasks, including but not limited to artistic style transfer, enhancement, colorization, image-to-image translation and intrinsic image decomposition.
...
...

References

SHOWING 1-10 OF 46 REFERENCES
Online Video Stream Abstraction and Stylization
TLDR
An automatic method for online video stream abstraction is given, producing a temporally coherent output video stream, in a style with large regions of constant color and highlighted bold edges, allowing the approach to be used for live communication and other related applications.
Artistic Style Transfer for Videos
TLDR
This work presents an approach that transfers the style from one image (for example, a painting) to a whole video sequence, and makes use of recent advances in style transfer in still images and proposes new initializations and loss functions applicable to videos.
Blind video temporal consistency
TLDR
This work proposes a gradient-domain technique that is blind to the particular image processing algorithm, and takes a series of processed frames that suffers from flickering and generates a temporally-consistent video sequence.
Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
TLDR
This work proposes an alternative approach that moves the computational burden to a learning stage and trains compact feed-forward convolutional networks to generate multiple samples of the same texture of arbitrary size and to transfer artistic style from a given image to any other image.
Region-based temporally consistent video post-processing
TLDR
This paper observes experimentally that many image/video enhancement algorithms enforce a spatially consistent prior on the enhancement, and segment each frame into several regions and temporally-spatially adjust the enhancement of regions of different frames, taking into account fidelity, temporal consistency and spatial consistency.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
TLDR
This work considers image transformation problems, and proposes the use of perceptual loss functions for training feed-forward networks for image transformation tasks, and shows results on image style transfer, where aFeed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time.
Deep Feature Flow for Video Recognition
TLDR
Deep feature flow is presented, a fast and accurate framework for video recognition that runs the expensive convolutional sub-network only on sparse key frames and propagates their deep feature maps to other frames via a flow field and achieves significant speedup as flow computation is relatively fast.
Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks
TLDR
This paper introduces a novel concept to augment such generative architectures with semantic annotations, either by manually authoring pixel labels or using existing solutions for semantic segmentation, resulting in a content-aware generative algorithm that offers meaningful control over the outcome.
DeepMovie: Using Optical Flow and Deep Neural Networks to Stylize Movies
TLDR
The main contribution of this paper is to use optical flow to initialize the style transfer optimization so that the texture features move with the objects in the video.
StyleBank: An Explicit Representation for Neural Image Style Transfer
TLDR
This work proposes StyleBank, which is composed of multiple convolution filter banks and each filter bank explicitly represents one style, for neural image style transfer, the first style transfer network that links back to traditional texton mapping methods, and hence provides new understanding on neural style transfer.
...
...