Interactive Image Segmentation with Latent Diversity

  title={Interactive Image Segmentation with Latent Diversity},
  author={Zhuwen Li and Qifeng Chen and Vladlen Koltun},
  journal={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},
Interactive image segmentation is characterized by multimodality. When the user clicks on a door, do they intend to select the door or the whole house? We present an end-to-end learning approach to interactive image segmentation that tackles this ambiguity. Our architecture couples two convolutional networks. The first is trained to synthesize a diverse set of plausible segmentations that conform to the user's input. The second is trained to select among these. By selecting a single solution… 

Figures and Tables from this paper

MultiSeg: Semantically Meaningful, Scale-Diverse Segmentations From Minimal User Input
MultiSeg is presented, a scale-diverse interactive image segmentation network that incorporates a set of two-dimensional scale priors into the model to generate aSet of scale-varying proposals that conform to the user input.
Rethinking Interactive Image Segmentation: Feature Space Annotation
This work proposes interactive and simultaneous segment annotation from multiple images guided by feature space projection and optimized by metric learning as the labeling progresses, and shows that this approach can surpass the accuracy of state-of-the-art methods in foreground segmentation datasets: iCoSeg, DAVIS, and Rooftop.
Interactive Image Segmentation via Backpropagating Refinement Scheme
An interactive image segmentation algorithm, which accepts user-annotations about a target object and the background, is proposed and the backpropagating refinement scheme (BRS) is developed, which corrects the mislabeled pixels in the initial result.
Localized Interactive Instance Segmentation
This work proposes a clicking scheme wherein user interactions are restricted to the proximity of the object, and a novel transformation of the user-provided clicks to generate a weak localization prior on the object which is consistent with image structures such as edges, textures etc.
Content-Aware Multi-Level Guidance for Interactive Instance Segmentation
This work proposes a novel transformation of user clicks to generate content-aware guidance maps that leverage the hierarchical structural information present in an image to outperform existing approaches that require state-of-the-art segmentation networks pre-trained on large scale segmentation datasets.
Interactive Image Segmentation With First Click Attention
The critical role of the first click about providing the location and main body information of the target object and a click-based loss function and a structural integrity strategy for better segmentation effect are demonstrated.
Scale-aware multi-level guidance for interactive instance segmentation
This work proposes a novel transformation of user clicks to generate scale-aware guidance maps that leverage the hierarchical structural information present in an image to outperform existing approaches that require state-of-the-art segmentation networks pre-trained on large scale segmentation datasets.
Interactive Full Image Segmentation by Considering All Regions Jointly
This work proposes an interactive, scribble-based annotation framework which operates on the whole image to produce segmentations for all regions, and adapt Mask-RCNN into a fast interactive segmentation framework and introduces an instance-aware loss measured at the pixel-level in the full image canvas, which lets predictions for nearby regions properly compete for space.
Two-in-One Refinement for Interactive Segmentation
This work proposes a simple yet intuitive two-in-one refinement strategy placing clicks on the boundary of the object of interest and proposes a boundary-aware loss that encourages segmentation masks to respect instance boundaries.


Deep Interactive Object Selection
This paper presents a novel deep-learning-based algorithm which has much better understanding of objectness and can reduce user interactions to just a few clicks and is superior to all existing interactive object selection approaches.
Geodesic Matting: A Framework for Fast Interactive Image and Video Segmentation and Matting
An interactive framework for soft segmentation and matting of natural images and videos is presented, based on the optimal, linear time, computation of weighted geodesic distances to user-provided scribbles, from which the whole data is automatically segmented.
Discriminative Re-ranking of Diverse Segmentations
This paper introduces a hybrid, two-stage approach to semantic image segmentation. In the first stage a probabilistic model generates a set of diverse plausible segmentations. In the second stage, a
Predicting Multiple Structured Visual Interpretations
This work leverages recent advances in the contextual submodular maximization literature to learn a sequence of predictors and empirically demonstrates the simplicity and performance of the approach on multiple challenging vision tasks.
Optimizing Expected Intersection-Over-Union with Candidate-Constrained CRFs
This work studies the question of how to make loss-aware predictions in image segmentation settings where the evaluation function is the Intersection-over-Union (IoU) measure and develops two new methods that draw ideas from both existing approaches.
Photographic Image Synthesis with Cascaded Refinement Networks
  • Qifeng Chen, V. Koltun
  • Computer Science
    2017 IEEE International Conference on Computer Vision (ICCV)
  • 2017
It is shown that photographic images can be synthesized from semantic layouts by a single feedforward network with appropriate structure, trained end-to-end with a direct regression objective.
"GrabCut": interactive foreground extraction using iterated graph cuts
A more powerful, iterative version of the optimisation of the graph-cut approach is developed and the power of the iterative algorithm is used to simplify substantially the user interaction needed for a given quality of result.
Intelligent scissors for image composition
Intelligent Sc scissors allows objects within digital images to be extracted quickly and accurately using simple gesture motions with a mouse, and allows creation of convincing compositions from existing images while dramatically increasing the speed and precision with which objects can be extracted.
Geodesic star convexity for interactive image segmentation
A new shape constraint for interactive image segmentation is introduced, an extension of Veksler's star-convexity prior, in two ways: from a single star to multiple stars and from Euclidean rays to Geodesic paths.
Fully Convolutional Networks for Semantic Segmentation
It is shown that convolutional networks by themselves, trained end- to-end, pixels-to-pixels, improve on the previous best result in semantic segmentation.