Sketch-based Creativity Support Tools using Deep Learning

  title={Sketch-based Creativity Support Tools using Deep Learning},
  author={Forrest Huang and Eldon Schoop and David R Ha and Jeffrey Nichols and John F. Canny},
Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines’ ability in understanding and generating visual content. An exciting area of development explores deep-learning approaches used to model human sketches, opening opportunities for creative applications. This chapter describes three fundamental steps in developing deep-learning-driven creativity support tools that consumes… 



How do humans sketch objects?

This paper is the first large scale exploration of human sketches, developing a bag-of-features sketch representation and using multi-class support vector machines, trained on the sketch dataset, to classify sketches.

CoSE: Compositional Stroke Embeddings

It is demonstrated qualitatively and quantitatively that the proposed model is able to model the appearance of individual strokes, as well as the compositional structure of larger diagram drawings, and is suitable for interactive use cases such as auto-completing diagrams.

SILK: sketching interfaces like krazy

  • J. Landay
  • Computer Science
    CHI Conference Companion
  • 1996
An interactive tool called SILK that allows designers to quickly sketch an interface using an electronic pad and stylus, which can be semi-automatically transformed into a complete, operational interface in a specified look-and-feel.

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

A novel network is designed that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels, and enriched using cross-modal co-attention and hierarchical node fusion at every level to form a better embedding space to conduct retrieval.

SketchyScene: Richly-Annotated Scene Sketches

The potential impact of SketchyScene is demonstrated by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc.

Apparition: Crowdsourced User Interfaces that Come to Life as You Sketch Them

This paper introduces crowdsourcing techniques and tools for prototyping interactive systems in the time it takes to describe the idea, and introduces Powering Apparition, the first self-coordinated, real-time crowdsourcing infrastructure.

PortraitSketch: face sketching assistance for novices

PortraitSketch is an interactive drawing system that helps novices create pleasing, recognizable face sketches without requiring prior artistic training and is designed to encourage a sense of ownership and accomplishment in the user.

SketchyCOCO: Image Generation From Freehand Scene Sketches

This work introduces the first method for automatic image generation from scene-level freehand sketches that allows for controllable image generation by specifying the synthesis goal via free hand sketches and builds a large-scale composite dataset called SketchyCOCO to support and evaluate the solution.

CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

This work develops a Collaborative image-Drawing game between two agents, called CoDraw, which is grounded in a virtual world that contains movable clip art objects and presents models for the task and benchmark them using both fully automated evaluation and by having them play the game live with humans.

Photo-Sketching: Inferring Contour Drawings From Images

Surprisingly, when the model fine-tunes on BSDS500, the model achieves the state-of-the-art performance in salient boundary detection, suggesting contour drawing might be a scalable alternative to boundary annotation, which at the same time is easier and more interesting for annotators to draw.