Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval

@article{Dey2019DoodleTS,
  title={Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval},
  author={Sounak Dey and Pau Riba and Anjan Dutta and Josep Llad{\'o}s and Yi-Zhe Song},
  journal={2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019},
  pages={2174-2183}
}
  • S. DeyPau Riba Yi-Zhe Song
  • Published 6 April 2019
  • Computer Science
  • 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
In this paper, we investigate the problem of zero-shot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. [] Key Method We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330,000 sketches and 204,000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets…

Figures and Tables from this paper

Style-Guided Zero-Shot Sketch-based Image Retrieval

This work proposes a novel framework which decomposes each image and sketch into its domainindependent content and a domain, as well as data-dependent variation/style component, and utilizes the image specific styles to guide the generation of fake images using the query content to be used for retrieval.

ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval

This work proposes an approach to jointly optimize sketch-to-photo synthesis and the image retrieval, which achieves state-of-the-art performance on two widely used ZS-SBIR datasets and surpasses previous methods by a large margin.

Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval

A novel Three-Stream Joint Training Network (3JOIN) for the ZS-SBIR task is proposed, which uses a teacher network to extract the implicit semantics of the samples without the aid of other semantics and transfer the learned knowledge to unseen classes.

StyleGuide: Zero-Shot Sketch-Based Image Retrieval Using Style-Guided Image Generation

This work proposes a detection approach for unseen class samples in the search-set, based on pre-computed seen class-prototypes, to obtain a refined search- set for a particular unseen-class query, resulting in improved retrieval performance.

More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval

This paper introduces a novel semi-supervised framework for cross-modal retrieval that can additionally leverage large-scale unlabelled photos to account for data scarcity, and treats generation and retrieval as two conjugate problems, where a joint learning procedure is devised for each module to mutually benefit from each other.

Zero-Shot Sketch Based Image Retrieval Using Graph Transformer

A novelgraph transformer based zero-shot sketch-based image retrieval (GTZSR) framework for solving ZS-SBIR tasks which uses a novel graph transformer to preserve the topology of the classes in the semantic space and propagates the context-graph of theclasses within the embedding features of the visual space.

Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval

A reinforcement learning based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode and introduces a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval.

Bi-Directional Domain Translation for Zero-Shot Sketch-Based Image Retrieval

A Bi-directional Domain Translation (BDT) framework is proposed for ZS-SBIR, in which the image domain and sketch domain can be translated to each other through disentangled structure and appearance features to facilitate structure-based retrieval.

Zero-shot sketch-based image retrieval with structure-aware asymmetric disentanglement

Edinburgh Research Explorer Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval

This paper devise a reinforcement learning based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode, and introduces a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval.
...

References

SHOWING 1-10 OF 42 REFERENCES

Sketch Me That Shoe

A deep tripletranking model for instance-level SBIR is developed with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data.

Fine-Grained Sketch-Based Image Retrieval by Matching Deformable Part Models

  • Li
  • Computer Science
  • 2014
This paper learns deformable part-based model (DPM) as a mid-level representation to discover and encode the various poses in sketch and image domains independently, after which graph matching is performed on DPMs to establish pose correspondences across the two domains.

Zero-Shot Sketch-Image Hashing

ZSIH is the first zero- shot hashing work suitable for SBIR and cross-modal search and forms a generative hashing scheme in reconstructing semantic knowledge representations for zero-shot retrieval.

Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval

This paper introduces a novel binary coding method, named Deep Sketch Hashing (DSH), where a semi-heterogeneous deep architecture is proposed and incorporated into an end-to-end binary coding framework, and is the first hashing work specifically designed for category-level SBIR with an end to end deep architecture.

Sketch based Image Retrieval using Learned KeyShapes (LKS)

This work presents a novel method for describing sketches based on detecting mid-level patterns called learned keyshapes and shows that this method allows us to achieve good performance even when the authors use around 20% of the sketch content.

SketchNet: Sketch Classification with Web Images

This study presents a weakly supervised approach that discovers the discriminative structures of sketch images, given pairs of Sketch images and web images, using a deep convolutional neural network, named SketchNet.

Sketch-based 3D shape retrieval using Convolutional Neural Networks

  • Fang WangLe KangYi Li
  • Computer Science
    2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2015
This work drastically reduces the number of views to only two predefined directions for the whole dataset, and learns two Siamese Convolutional Neural Networks, one for the views and one forThe sketches, which is significantly better than state of the art approaches, and outperforms them in all conventional metrics.

A performance evaluation of gradient field HOG descriptor for sketch based image retrieval

Generative Domain-Migration Hashing for Sketch-to-Image Retrieval

A Generative Domain-migration Hashing approach, which for the first time generates hashing codes from synthetic natural images that are migrated from sketches that can migrate sketches to their indistinguishable image counterparts while preserving the domain-invariant information of sketches.

How do humans sketch objects?

This paper is the first large scale exploration of human sketches, developing a bag-of-features sketch representation and using multi-class support vector machines, trained on the sketch dataset, to classify sketches.