• Corpus ID: 232097550

On Disentangled Representations Learned from Correlated Data

@inproceedings{Trauble2021OnDR,
  title={On Disentangled Representations Learned from Correlated Data},
  author={Frederik Trauble and Elliot Creager and Niki Kilbertus and Francesco Locatello and Andrea Dittadi and Anirudh Goyal and Bernhard Scholkopf and Stefan Bauer},
  booktitle={ICML},
  year={2021}
}
The focus of disentanglement approaches has been on identifying independent factors of variation in data. However, the causal variables underlying real-world observations are often not statistically independent. In this work, we bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data in a large-scale empirical study (including 4260 models). We show and quantify that systematically induced correlations in the dataset… 
Disentangling Embedding Spaces with Minimal Distributional Assumptions
TLDR
This work links conceptual explainability for vision models with disentanglement learning and ICA, and derives the disjoint attributions (DA) concept discovery method that is applicable to a broader class of problems than current approaches, but yet possesses a formal identifiability guarantee.
CITRIS: Causal Identifiability from Temporal Intervened Sequences
TLDR
This work proves identifiability in a more general setting, in which only some components of a causal factor are affected by interventions, and extends CITRIS to leverage and disentangle representations obtained by already pretrained autoencoders.
On Causally Disentangled Representations
TLDR
This work motivates the need for new metrics and datasets to study causal disentanglement and proposes two evaluation metrics and a dataset that capture the desiderata of disentangled causal process and performs an empirical study on state of the artdisentangled representation learners using the metrics and dataset to evaluate them from causal perspective.
Surrogate Model for Field Optimization Using Beta-VAE Based Regression
TLDR
Beta-VAE based regression is proposed to generate simulation surrogates for use in optimization workflow in which factorized representation in latent space makes the model interpretable while probabilistic layers determine the prediction uncertainty.
Equivariant Disentangled Transformation for Domain Generalization under Combination Shift
TLDR
Experimental results demonstrate that invariance may be insufficient, and it is important to exploit the equivariance structure in the combination shift problem, and a simple yet effective method is derived, referred to as equivariant disentangled transformation (EDT), which augments the data based on the algebraic structures of labels and makes the transformation satisfy the Equivariance and disentanglement requirements.
Discrete Key-Value Bottleneck
TLDR
A model architecture is proposed, building upon a discrete bottleneck containing pairs of separate and learnable (key, value) codes, that reduces the complexity of the hypothesis class and reduces the common vulnerability to non-i.i.d. and non-stationary training distributions.
Overcoming Shortcut Learning in a Target Domain by Generalizing Basic Visual Factors from a Source Domain
TLDR
The main finding is that leveraging data from a source domain is an effective way to mitigate shortcut learning and networks can learn to consider only predictive factors and ignore potential shortcut factors during inference.
Causal Graphs Underlying Generative Models: Path to Learning with Limited Data
TLDR
This work provides a simple algorithm that relies on perturbation experiments on latent codes of a pre-trained generative autoencoder to uncover a causal graph that is implied by the generative model and demonstrates that the learnt causal graph from the algorithm can be used to predict a specific property for sequences which are unseen.
Learning Robust Representation for Joint Grading of Ophthalmic Diseases via Adaptive Curriculum and Feature Disentanglement
TLDR
A framework for joint grading with the addition of DAW and DETACH is proposed, which learns robust disentangled feature representations to explore internal correlations between DR and DME and achieve better grading performance.
Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning
TLDR
This work formalizes the concept of underspecification and proposes a method to identify and partially address it, and trains multiple models with an independence constraint that forces them to implement different functions, resulting in a global model with superior OOD performance.
...
...

References

SHOWING 1-10 OF 69 REFERENCES
Interventional Robustness of Deep Latent Variable Models
TLDR
The interventional robustness score is introduced, which provides a quantitative evaluation of robustness in learned representations with respect to interventions on generative factors and changing nuisance factors, and how this score can be estimated from labeled observational data, that may be confounded, and further provide an efficient algorithm that scales linearly in the dataset size.
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
TLDR
This work considers the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and proposes a variational inference based approach to inferdisentangled latent factors.
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
TLDR
This paper theoretically shows that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data, and trains more than 12000 models covering most prominent methods and evaluation metrics on seven different data sets.
Weakly Supervised Disentanglement with Guarantees
TLDR
A theoretical framework is provided to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching and empirically verify the guarantees and limitations of several weak supervision methods.
Disentangling Factors of Variation Using Few Labels
TLDR
Overall, this paper empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.
On the Fairness of Disentangled Representations
TLDR
Analyzing the representations of more than 10,000 trained state-of-the-art disentangled models, it is observed that several disentanglement scores are consistently correlated with increased fairness, suggesting that disENTanglement may be a useful property to encourage fairness when sensitive variables are not observed.
Isolating Sources of Disentanglement in Variational Autoencoders
We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate our $\beta$-TCVAE (Total Correlation
A Framework for the Quantitative Evaluation of Disentangled Representations
TLDR
A framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available is proposed and three criteria are explicitly defined and quantified to elucidate the quality of learnt representations and thus compare models on an equal basis.
Evaluating the Disentanglement of Deep Generative Models through Manifold Topology
TLDR
This work presents a method for quantifying disentanglement that only uses the generative model, by measuring the topological similarity of conditional submanifolds in the learned representation and finds that this method ranks models similarly to existing methods.
Learning Disentangled Representation with Pairwise Independence
TLDR
This work proposes a new method based on a pairwise independence assumption to learn the disentangled representation and shows that the proposed method gives competitive performances as compared with other state-of-the-art methods.
...
...