SketchParse: Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks

@article{Sarvadevabhatla2017SketchParseTR,
  title={SketchParse: Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks},
  author={Ravi Kiran Sarvadevabhatla and Isht Dwivedi and Abhijat Biswas and Sahil Manocha and R. VenkateshBabu},
  journal={Proceedings of the 25th ACM international conference on Multimedia},
  year={2017}
}
The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SKETCHPARSE, the first deep-network architecture for fully automatic parsing of freehand object sketches. SKETCHPARSE is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing… 

Figures and Tables from this paper

SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation
TLDR
This paper proposes an approach for multi-class sketch semantic segmentation by considering it as a sequence-to-sequence generation problem, and presents an end- to-end learned network, SketchSegNet+, built on recurrent neural networks (RNN), to translate a sequence of strokes into a sequences of semantic labels.
Deep Semantic Parsing of Freehand Sketches With Homogeneous Transformation, Soft-Weighted Loss, and Staged Learning
TLDR
A novel deep framework for part-level semantic parsing of freehand sketches makes three main contributions that are experimentally shown to have substantial practical merit, and integrates the proposed three methods into a unified framework of deep semantic sketch parsing (DeepSSP).
Sketchsegnet: A Rnn Model for Labeling Sketch Strokes
TLDR
This paper treats the problem of stroke-level sketch segmentation as a seqence-to-sequence generation problem, and a reccurent nueral networks (RNN)-based model SketchSegNet is presented to translate sequence of strokes into thier semantic part labels.
Fast Sketch Segmentation and Labeling With Deep Learning
TLDR
A deep neural network is trained to transfer existing segmentations and labelings from three-dimensional models to freehand sketches without requiring numerous well-annotated sketches as training data, enabling further integration in interactive drawing systems.
SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks
TLDR
This work introduces SketchGNN, a convolutional graph neural network for semantic segmentation and labeling of freehand vector sketches that significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentsation and has magnitudes fewer parameters than both image-based and sequence-based methods.
Fast Sketch Segmentation and Labeling with Deep Learning
  • Lei Li
  • Computer Science
  • 2018
TLDR
A deep neural network is trained to transfer existing segmentations and labelings from 3D models to freehand sketches without requiring numerous well-annotated sketches as training data, enabling further integration in interactive drawing systems.
AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches
TLDR
AI-Sketcher, a deep generative model for generating high-quality multiclass sketches is introduced, which improves drawing quality by employing a CNN-based autoencoder and introduces an influence layer to more precisely guide the generation of each stroke by directly referring to the training data.
2D freehand sketch labeling using CNN and CRF
TLDR
A novel hybrid approach for fast automatic sketch labeling, which is based on convolutional neural network (CNN) and conditional random field (CRF), which achieves the state-of-the-art level on both accuracy and runtime.
SketchGCN: Semantic Sketch Segmentation with Graph Convolutional Networks
TLDR
SketchGCN significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentation and labeling of free-hand sketches and has magnitudes fewer parameters than both image-based and sequence-based methods.
...
...

References

SHOWING 1-10 OF 59 REFERENCES
Sketch Me That Shoe
TLDR
A deep tripletranking model for instance-level SBIR is developed with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data.
Enabling My Robot To Play Pictionary: Recurrent Neural Networks For Sketch Recognition
TLDR
This work introduces a Gated Recurrent Unit based framework which leverages deep sketch features and weighted per-timestep loss to achieve state-of-the-art results on a large database of freehand object sketches across a large number of object categories.
Sketch-a-Net that Beats Humans
TLDR
A multi-scale multi-channel deep neural network framework that yields sketch recognition performance surpassing that of humans, and not only delivers the best performance on the largest human sketch dataset to date, but also is small in size making efficient training possible using just CPUs.
The sketchy database
TLDR
The Sketchy database is presented, the first large-scale collection of sketch-photo pairs and it is shown that the learned representation significantly outperforms both hand-crafted features as well as deep features trained for sketch or photo classification.
Fine-Grained Sketch-Based Image Retrieval by Matching Deformable Part Models
  • Li
  • Computer Science
  • 2014
TLDR
This paper learns deformable part-based model (DPM) as a mid-level representation to discover and encode the various poses in sketch and image domains independently, after which graph matching is performed on DPMs to establish pose correspondences across the two domains.
Joint Object and Part Segmentation Using Deep Learned Potentials
TLDR
This paper proposes a joint solution that tackles semantic object and part segmentation simultaneously, in which higher object-level context is provided to guide part segmentsation, and more detailed part-level localization is utilized to refine object segmentation.
Instance-Aware Semantic Segmentation via Multi-task Network Cascades
  • Jifeng Dai, Kaiming He, Jian Sun
  • Computer Science
    2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2016
TLDR
This paper presents Multitask Network Cascades for instance-aware semantic segmentation, which consists of three networks, respectively differentiating instances, estimating masks, and categorizing objects, and develops an algorithm for the nontrivial end-to-end training of this causal, cascaded structure.
Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net
TLDR
This work proposes a "Hierarchical Auto-Zoom Net" (HAZN) for object part parsing which adapts to the local scales of objects and parts and significantly outperforms the state-of-the-arts by 5% mIOU and is especially better at segmenting small instances and small parts.
Example-Based Sketch Segmentation and Labeling Using CRFs
TLDR
This work introduces a new approach for segmentation and label transfer in sketches that substantially improves the state of the art, and uses a Conditional Random Field to find the most probable global configuration.
How do humans sketch objects?
TLDR
This paper is the first large scale exploration of human sketches, developing a bag-of-features sketch representation and using multi-class support vector machines, trained on the sketch dataset, to classify sketches.
...
...