• Corpus ID: 235727348

Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning

@article{Ke2021SystematicEO,
  title={Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning},
  author={Nan Rosemary Ke and Aniket Didolkar and Sarthak Mittal and Anirudh Goyal and Guillaume Lajoie and Stefan Bauer and Danilo Jimenez Rezende and Yoshua Bengio and Michael C. Mozer and Christopher Joseph Pal},
  journal={ArXiv},
  year={2021},
  volume={abs/2107.00848}
}
Inducing causal relationships from observations is a classic problem in machine learning. Most work in causality starts from the premise that the causal variables themselves are observed. However, for AI agents such as robots trying to make sense of their environment, the only observables are low-level variables like pixels in images. To generalize well, an agent must induce high-level variables, particularly those which are causal or are affected by causal variables. A central goal for AI and… 
Learning Causal Overhypotheses through Exploration in Children and Computational Models
TLDR
There are significant differences between information-gain optimal RL exploration in causal environments and the exploration of children in the same environments, and this work introduces a novel RL environment designed with a controllable causal structure, which allows to evaluate exploration strategies used by both agents and children in a unified environment.
Learning to Induce Causal Structure
TLDR
This work designs a neural network architecture that learns the mapping from both observational and interventional data to graph structures via supervised training on synthetic graphs and shows that the proposed model generalizes not only to new synthetic graphs but also to naturalistic graphs.
Neural Production Systems: Learning Rule-Governed Visual Dynamics
TLDR
This work takes inspiration from cognitive science and resurrects a classic approach, production systems, which consist of a set of rule templates that are applied by binding placeholder variables in the rules to specific entities, which achieves a flexible, dynamic flow of control and serves to factorize entity-specific and rule-based information.
The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents
TLDR
It is observed that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case, and that the generalization performance of a simple downstream proxy task reliably predicts the generalized performance of the authors' RL agents under a wide range of OOD settings.
The Role of Pretrained Representations for the OOD Generalization of RL Agents
TLDR
This work evaluates to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents, and finds that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case.
Representation Learning for Out-Of-Distribution Generalization in Reinforcement Learning
TLDR
This paper aims to establish the first systematic characterization of the usefulness of learned representations for real-world OOD downstream tasks by training over 10,000 reinforcement learning policies to what extent different representation properties affect out-of-distribution (OOD) generalization.
T HE R OLE OF P RETRAINED R EPRESENTATIONS FOR THE OOD G ENERALIZATION OF RL A GENTS
TLDR
It is observed that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case, and that the generalization performance of a simple downstream proxy task reliably predicts the generalized performance of the authors' RL agents under a wide range of OOD settings.
The Sandbox Environment for Generalizable Agent Research (SEGAR)
TLDR
The Sandbox Environment for Generalizable Agent Research (SEGAR) is built, which improves the ease and accountability of generalization research in RL, as generalization objectives can be easy designed by specifying task distributions, which in turns allows the researcher to measure the nature of the generalization objective.
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA
TLDR
A rigorous identifiability theory is developed, building on recent nonlinear independent component analysis (ICA) results, that shows how the latent variables can be recovered up to permutation if one regularizes the latent mechanisms to be sparse and if some graph connectivity criterion is satisfied by the data generating process.
Compositional Attention: Disentangling Search and Retrieval
TLDR
This work proposes a novel attention mechanism, called Compositional Attention, that replaces the standard head structure, and demonstrates that it outperforms standard multi-head attention on a variety of tasks, including some out-of-distribution settings.
...
1
2
...

References

SHOWING 1-10 OF 58 REFERENCES
Causal Induction from Visual Observations for Goal Directed Tasks
TLDR
This work proposes to endow an artificial agent with the capability of causal reasoning for completing goal-directed tasks and develops learning-based approaches to inducing causal knowledge in the form of directed acyclic graphs, which can be used to contextualize a learned goal-conditional policy to perform tasks in novel environments with latent causal structures.
ACRE: Abstract Causal REasoning Beyond Covariation
TLDR
This work introduces the Abstract Causal REasoning (ACRE) dataset for systematic evaluation of current vision systems in causal induction and notices that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning.
Causal Reasoning from Meta-reinforcement Learning
TLDR
It is suggested that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here, and this work offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.
Toward Causal Representation Learning
TLDR
Fundamental concepts of causal inference are reviewed and related to crucial open problems of machine learning, including transfer and generalization, thereby assaying how causality can contribute to modern machine learning research.
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
TLDR
This work presents a novel method that learns to discover objects and model their physical interactions from raw visual images in a purely unsupervised fashion and incorporates prior knowledge about the compositional nature of human perception to factor interactions between object-pairs and learn efficiently.
Amortized learning of neural causal representations
TLDR
A novel algorithm is represented called CRN for learning causal models using neural networks that could scale much better with the number of variables and take in previously learned information to facilitate learning of new causal models.
Entity Abstraction in Visual Model-Based Reinforcement Learning
TLDR
Object-centric perception, prediction, and planning (OP3), which is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses them to predict and plan, is presented.
Causal Confusion in Imitation Learning
TLDR
It is shown that causal misidentification occurs in several benchmark control domains as well as realistic driving settings, and the proposed solution to combat it through targeted interventions to determine the correct causal model is validated.
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
TLDR
This work proposes to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities and shows that causal structures can be parameterized via continuous variables and learned end-to-end.
Causally Correct Partial Models for Reinforcement Learning
TLDR
This paper shows that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning, and introduces a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.
...
1
2
3
4
5
...