Class-Based Styling: Real-Time Localized Style Transfer with Semantic Segmentation

  title={Class-Based Styling: Real-Time Localized Style Transfer with Semantic Segmentation},
  author={Lironne Kurzman and David V{\'a}zquez and Issam H. Laradji},
  journal={2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)},
We propose a Class-Based Styling method (CBS) that can map different styles for different object classes in real-time. CBS achieves real-time performance by carrying out two steps simultaneously. While a semantic segmentation method is used to obtain the mask of each object class in a video frame, a styling method is used to style that frame globally. Then an object class can be styled by combining the segmentation mask and the styled image. The user can also select multiple styles so that… 

Figures from this paper

A Method for Arbitrary Instance Style Transfer
A topologically inspired algorithm called Forward Stretching is proposed to tackle the problem of transferring style to an instance in an arbitrary shape by transforming an instance into a tensor representation, which allows us to transfer style to this instance itself directly.
SelfieArt: Interactive Multi-Style Transfer for Selfies and Videos with Soft Transitions
We introduce SelfieArt, an interactive technique for performing multi-style transfer for portraits and videos. Our method provides a simple and intuitive way of producing exquisite artistic results
What Ails One-Shot Image Segmentation: A Data Perspective
A new evaluation dataset Tiered One-shot Segmentation (TOSS) based on the PASCAL 5 and FSS-1000 datasets, and associated metrics for each tier is proposed, which enforces consistent accuracy measurement for existing methods, and affords fine-grained insights into the applicability of a method to real applications.
Style Transfer applied to CT image downscaling: a study case from Brazilian Coquinas
The identification of micropore systems in carbonate rocks is an important task of image processing because of the high impact these systems cause on fluid flow. Currently, one of the main tools used
Interactive Style Transfer: All is Your Palette
—Neural style transfer (NST) can create impressive artworks by transferring reference style to content image. Current image-to-image NST methods are short of fine-grained controls, which are often
GAN Memory with No Forgetting
This work proposes a GAN memory that is capable of realistically remembering a stream of generative processes with \emph{no} forgetting and demonstrates the superiority of the method over existing approaches and its effectiveness in alleviating catastrophic forgetting for lifelong classification problems.


Son of Zorn's lemma: Targeted style transfer using instance-aware semantic segmentation
A method for targeted style transfer that simultaneously segments and stylizes single objects selected by the user, and uses a Markov random field model to smooth and anti-alias outlier pixels near object boundaries, so that stylized objects naturally blend into their surroundings.
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
This work considers image transformation problems, and proposes the use of perceptual loss functions for training feed-forward networks for image transformation tasks, and shows results on image style transfer, where aFeed-forward network is trained to solve the optimization problem proposed by Gatys et al. in real-time.
Neural Style Transfer: A Review
A taxonomy of current algorithms in the field of NST is proposed and several evaluation methods are presented and compared to compare different NST algorithms both qualitatively and quantitatively.
DABNet: Depth-wise Asymmetric Bottleneck for Real-time Semantic Segmentation
A novel Depthwise Asymmetric Bottleneck Bottleneck (DAB) module is proposed to address this dilemma, which efficiently adopts depth-wise asymmetric convolution and dilated convolution to build a bottleneck structure.
Mask R-CNN
This work presents a conceptually simple, flexible, and general framework for object instance segmentation, which extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition.
The Cityscapes Dataset for Semantic Urban Scene Understanding
This work introduces Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling, and exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity.
A Neural Algorithm of Artistic Style
This work introduces an artificial system based on a Deep Neural Network that creates artistic images of high perceptual quality and offers a path forward to an algorithmic understanding of how humans create and perceive artistic imagery.
Very Deep Convolutional Networks for Large-Scale Image Recognition
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Automatic differentiation in PyTorch
An automatic differentiation module of PyTorch is described — a library designed to enable rapid research on machine learning models that focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead.