Fully Convolutional Instance-Aware Semantic Segmentation

  title={Fully Convolutional Instance-Aware Semantic Segmentation},
  author={Yi Li and Haozhi Qi and Jifeng Dai and Xiangyang Ji and Yichen Wei},
  journal={2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  • Yi LiHaozhi Qi Yichen Wei
  • Published 23 November 2016
  • Computer Science
  • 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
We present the first fully convolutional end-to-end solution for instance-aware semantic segmentation task. It inherits all the merits of FCNs for semantic segmentation [29] and instance mask proposal [5]. It performs instance mask prediction and classification jointly. The underlying convolutional representation is fully shared between the two sub-tasks, as well as between all regions of interest. The network architecture is highly integrated and efficient. It achieves state-of-the-art… 

Figures and Tables from this paper

ISDA: Position-Aware Instance Segmentation with Deformable Attention

A novel end-to-end instance segmentation method termed ISDA reshapes the task into predicting a set of object masks, which are generated via traditional convolution operation with learned position-aware kernels and features of objects.

DASNet: Reducing Pixel-level Annotations for Instance and Semantic Segmentation

This work aims to achieve high quality instance and semantic segmentation results over a small set of pixel-level mask annotations and a large set of box annotations and demonstrates substantially improved performance compared to existing semi-supervised approaches on PASCAL VOC 2012 dataset.

BiSeg: Simultaneous Instance Segmentation and Semantic Segmentation with Fully Convolutional Networks

The idea of position-sensitive score maps used in recent methods to a fusion of multiple score maps at different scales and partition modes is extended, and adopted as a robust likelihood for instance segmentation inference.

Learning Saliency Propagation for Semi-Supervised Instance Segmentation

This work proposes ShapeProp, which learns to activate the salient regions within the object detection and propagate the areas to the whole instance through an iterative learnable message passing module, which establishes new states of the art for semi-supervised instance segmentation.

Dense Convolutional Networks for Semantic Segmentation

The dense convolution unit (DCU) is proposed, which is more suitable for pixel-wise classification and serves as the classification layer and is a better option than standard convolution in FCNs.

ISTR: End-to-End Instance Segmentation with Transformers

This paper proposes an instance segmentation Transformer, termed ISTR, which is the first end-to-end framework of its kind, and demonstrates state-of-the-art performance even with approximation-based suboptimal embeddings.

Dense Convolution for Semantic Segmentation

A dense convolution scheme that is more suitable for semantic segmentation by producing the same number of output as its input and introducing spatial overlaps into current convolutions.

Fast Single Shot Instance Segmentation

The SSD-like detection structure and two-path architecture which can generate more accurate segmentation prediction without heavy calculation burden are adopted and accelerate the instance segmentation process with competitive result compared to MaskRCNN.

Semantic Instance Segmentation via Deep Metric Learning

A new method for semantic instance segmentation is proposed, by first computing how likely two pixels are to belong to the same object, and then by grouping similar pixels together, based on a deep, fully convolutional embedding model.

Deep Variational Instance Segmentation

A novel algorithm that directly utilizes a fully convolutional network (FCN) to predict instance labels and extends the classical Mumford-Shah variational segmentation problem to be able to handle permutation-invariant labels in the ground truth of instance segmentation.



Instance-Aware Semantic Segmentation via Multi-task Network Cascades

  • Jifeng DaiKaiming HeJian Sun
  • Computer Science
    2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2016
This paper presents Multitask Network Cascades for instance-aware semantic segmentation, which consists of three networks, respectively differentiating instances, estimating masks, and categorizing objects, and develops an algorithm for the nontrivial end-to-end training of this causal, cascaded structure.

Instance-Sensitive Fully Convolutional Networks

This paper develops FCNs that are capable of proposing instance-level segment candidates that do not have any high-dimensional layer related to the mask resolution, but instead exploits image local coherence for estimating instances.

Proposal-Free Network for Instance-Level Object Segmentation

A Proposal-Free Network (PFN) is proposed to address the instance-level object segmentation problem, which outputs the numbers of instances of different categories and the pixel-level information on i) the coordinates of the instance bounding box each pixel belongs to, and ii) the confidences ofDifferent categories for each pixel, based on pixel-to-pixel deep convolutional neural network.

Fully convolutional networks for semantic segmentation

The key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning.

Attention to Scale: Scale-Aware Semantic Image Segmentation

An attention mechanism that learns to softly weight the multi-scale features at each pixel location is proposed, which not only outperforms averageand max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales.

Learning Deconvolution Network for Semantic Segmentation

A novel semantic segmentation algorithm by learning a deep deconvolution network on top of the convolutional layers adopted from VGG 16-layer net, which demonstrates outstanding performance in PASCAL VOC 2012 dataset.

Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network

A novel weakly-supervised semantic segmentation algorithm based on Deep Convolutional Neural Network based on DCNN that exploits auxiliary segmentation annotations available for different categories to guide segmentations on images with only image-level class labels.

Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation

This work shows how to improve semantic segmentation through the use of contextual information, specifically, ' patch-patch' context between image regions, and 'patch-background' context, and formulate Conditional Random Fields with CNN-based pairwise potential functions to capture semantic correlations between neighboring patches.

Simultaneous Detection and Segmentation

This work builds on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN), introducing a novel architecture tailored for SDS, and uses category-specific, top-down figure-ground predictions to refine the bottom-up proposals.

Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform

This work proposes replacing the fully-connected CRF with domain transform (DT), a modern edge-preserving filtering method in which the amount of smoothing is controlled by a reference edge map, and shows that it yields comparable semantic segmentation results, accurately capturing object boundaries.