Structured Semantic Transfer for Multi-Label Recognition with Partial Labels

  title={Structured Semantic Transfer for Multi-Label Recognition with Partial Labels},
  author={Tianshui Chen and Tao Pu and Hefeng Wu and Yuan Xie and Liang Lin},
Multi-label image recognition is a fundamental yet practical task because real-world images inherently possess multiple semantic labels. However, it is difficult to collect large-scale multi-label annotations due to the complexity of both the input images and output label spaces. To reduce the annotation cost, we propose a structured semantic transfer (SST) framework that enables training multi-label recognition models with partial labels, i.e., merely some labels are known while other labels… 

Figures and Tables from this paper

Heterogeneous Semantic Transfer for Multi-label Recognition with Partial Labels
This work proposes a novel heterogeneous semantic transfer (HST) framework that consists of two complementary transfer modules that explore both within-image and cross-image semantic correlations to transfer the knowledge possessed by known labels to generate pseudo labels for the unknown labels.
Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels
A unified semantic-aware representation blending framework that exploits instance-level and prototype-level semantic representation to complement unknown labels by two complementary modules that obtains superior performance over current leading competitors on all known label proportion settings.
DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations
The strong alignment of textual and visual features pretrained with millions of auxiliary image-text pairs is utilized and Dual Context Optimization (DualCoOp) is proposed as a unified framework for partial-label MLR and zero-shot MLR.
Semantic Representation and Dependency Learning for Multi-Label Image Recognition
A novel and e-ective semantic representation and dependency learning (SRDL) framework to learn category-specific semantic representation for each category and capture semantic dependency among all categories.


Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition
A Semantic-Specific Graph Representation Learning (SSGRL) framework that consists of a semantic decoupling module that incorporates category semantics to guide learning semantic-specific representations and a semantic interaction module that correlates these representations with a graph built on the statistical label co-occurrence and explores their interactions via a graph propagation mechanism.
Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition
A knowledge-guided graph routing (KGGR) framework, which unifies prior knowledge of statistical label correlations with deep neural networks and can facilitate exploiting the information of correlated labels to help train better classifiers, especially for labels with limited training samples.
Learning a Deep ConvNet for Multi-Label Classification With Partial Labels
This work proposes to train a model with partial labels i.e. only some labels are known per image, and introduces a new classification loss that exploits the proportion of known labels per example.
Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition
A recurrent attention reinforcement learning framework to iteratively discover a sequence of attentional and informative regions that are related to different semantic objects and further predict label scores conditioned on these regions to facilitate multi-label recognition.
Exploit Bounding Box Annotations for Multi-Label Object Recognition
This paper first extracts object proposals from each image, then proposes to make use of ground-truth bounding box annotations (strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline.
CNN-RNN: A Unified Framework for Multi-label Image Classification
The proposed CNN-RNN framework learns a joint image-label embedding to characterize the semantic label dependency as well as the image- label relevance, and it can be trained end-to-end from scratch to integrate both information in a unified framework.
ML-MG: Multi-label Learning with Missing Labels Using a Mixed Graph
This work proposes a unified model of label dependencies by constructing a mixed graph, which jointly incorporates (i) instance-level similarity and class co-occurrence as undirected edges and (ii) semantic label hierarchy as directed edges.
Interactive Multi-Label CNN Learning With Partial Labels
A new loss function is introduced that regularizes the cross-entropy loss with a cost function that measures the smoothness of labels and features of images on the data manifold and allows to learn few informative similarities only for images in each mini-batch and handles changing feature representations.
Multi-label Image Recognition by Recurrently Discovering Attentional Regions
This work achieves the interpretable and contextualized multi-label image classification by developing a recurrent memorized-attention module that demonstrates superior performances over other existing state-of-the-arts in both accuracy and efficiency.
Matrix Completion for Multi-label Image Classification
This paper formulates image categorization as a multi-label classification problem using recent advances in matrix completion and proposes two convex algorithms for matrix completion based on a Rank Minimization criterion specifically tailored to visual data, and proves its convergence properties.