Taskonomy: Disentangling Task Transfer Learning

@article{Zamir2018TaskonomyDT,
  title={Taskonomy: Disentangling Task Transfer Learning},
  author={Amir Roshan Zamir and Alexander Sax and Bokui (William) Shen and Leonidas J. Guibas and Jitendra Malik and Silvio Savarese},
  journal={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2018},
  pages={3712-3722}
}
Do visual tasks have a relationship, or are they unrelated. [] Key Method This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. We provide a set of tools for computing and probing this…
Task2Vec: Task Embedding for Meta-Learning
TLDR
A method to generate vectorial representations of visual classification tasks which can be used to reason about the nature of those tasks and their relations, and is demonstrated to be capable of predicting task similarities that match the authors' intuition about semantic and taxonomic relations between different visual tasks.
Auto-Lambda: Disentangling Dynamic Task Relationships
TLDR
This work proposes a gradient-based meta learning framework, named Auto-λ, which explores continuous, dynamic task relationships via task-specific weightings, and can optimise any choice of combination of tasks through the formulation of a meta-loss; where the validation loss automatically influences task weightings throughout training.
Universal Representations : TowardsMulti-Task Learning & Beyond
TLDR
A novel weighting scheme, which adapts the task weighting over time by considering the rate of change of the loss for each task, and an auxiliary learning framework which can automatically generate knowledge to improve generalisation are presented.
Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing
TLDR
This paper proposes an attention-based architecture named Cerberus and a tailored training framework that effectively addresses aforementioned challenges and achieves state-of-the-art performance on all three tasks and shows concept affinity consistent with human cognition.
Learnable Parameter Similarity
TLDR
A novel LPS method that learns an effective metric to measure the similarity of second-order semantics hidden in trained models and creates a model set called ModelSet500 as a parameter similarity learning benchmark that contains 500 trained models.
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
TLDR
This model can effectively leverage its experience from one source task and apply it to multiple target tasks and enables zero-shot experience learning, whereby it can solve the target tasks without receiving any task-specific interactive training.
Learning Task Relatedness in Multi-Task Learning for Images in Context
TLDR
This work introduces Selective Sharing, a method that learns the inter-task relatedness from secondary latent features while the model trains and can automatically group tasks and allow them to share knowledge in a mutually beneficial way.
Deep Model Transferability from Attribution Maps
TLDR
An embarrassingly simple yet very efficacious approach to estimating the transferability of deep networks, especially those handling vision tasks, via projecting deep networks into a model space, wherein each network is treated as a point and the distances between two points are measured by deviations of their produced attribution maps.
Learning Across Tasks and Domains
TLDR
A novel adaptation framework that can operate across both task and domains and is complementary to existing domain adaptation techniques and extends them to cross tasks scenarios providing additional performance gains is introduced.
PROGRESSIVE MODULE NETWORKS
TLDR
By evaluating the reasoning process using human judges, this work shows that the proposed model effectively combines previous skill-sets, does not suffer from forgetting, and is fully differentiable.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 159 REFERENCES
Taskonomy: Disentangling Task Transfer Learning
TLDR
This work proposes a fully computational approach for modeling the transfer learning structure of the space of visual tasks via finding transfer learning dependencies across tasks in a dictionary of twenty-six 2D, 2.5D, 3D, and semantic tasks, and exploits it to reduce the demand for labeled data.
Multi-task Self-Supervised Visual Learning
TLDR
The results show that deeper networks work better, and that combining tasks—even via a na¨ýve multihead architecture—always improves performance.
A notion of task relatedness yielding provable multiple-task learning guarantees
TLDR
This work provides a formal framework for this notion of task relatedness, which captures a sub-domain of the wide scope of issues in which one may apply a multiple task learning approach and allows the formal derivation of generalization bounds that are strictly stronger than the previously known bounds.
Transitive Invariance for Self-Supervised Visual Representation Learning
TLDR
This paper proposes to generate a graph with millions of objects mined from hundreds of thousands of videos and argues to organize and reason the data with multiple variations to exploit different self-supervised approaches to learn representations invariant to inter-instance variations.
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
TLDR
This work finds that using a mid-level perception confers significant advantages over training end-to-end from scratch (i.e. not leveraging priors) in navigation-oriented tasks and develops an efficient max-coverage feature set that can be adopted in lieu of raw images.
Integrated perception with recurrent multi-task neural networks
TLDR
This work proposes a new architecture, which it calls "MultiNet", in which not only deep image features are shared between tasks, but where tasks can interact in a recurrent manner by encoding the results of their analysis in a common shared representation of the data.
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
TLDR
DeCAF, an open-source implementation of deep convolutional activation features, along with all associated network parameters, are released to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.
Learning to Learn
TLDR
This chapter discusses Reinforcement Learning with Self-Modifying Policies J. Schmidhuber, et al., and theoretical Models of Learning to Learn J. Baxter, a first step towards Continual Learning.
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
TLDR
A novel unsupervised learning approach to build features suitable for object detection and classification and to facilitate the transfer of features to other tasks, the context-free network (CFN), a siamese-ennead convolutional neural network is introduced.
Representation Learning by Learning to Count
TLDR
This paper uses two image transformations in the context of counting: scaling and tiling to train a neural network with a contrastive loss that produces representations that perform on par or exceed the state of the art in transfer learning benchmarks.
...
1
2
3
4
5
...