Invariant Representations through Adversarial Forgetting

@inproceedings{Jaiswal2019InvariantRT,
  title={Invariant Representations through Adversarial Forgetting},
  author={Ayush Jaiswal and Daniel Moyer and Greg Ver Steeg and Wael AbdAlmageed and P. Natarajan},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2019}
}
We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves state-of-the-art performance at learning invariance in both nuisance and bias settings on… 
[PDF] Semantic Reader
31 Citations
Highly Influential Citations
3
Background Citations
19
Methods Citations
13

Figures and Tables from this paper

31 Citations

mask-Net: Learning Context Aware Invariant Features using Adversarial Forgetting (Student Abstract)

This work proposes a novel approach to induce invariance using adversarial forgetting (AF), and initial experiments on learning invariant features such as accent on the STT task achieve better generalizations compared to the traditional models.

CONTRASTIVE REPRESENTATION LEARNING

A training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation is introduced, and a change to how test time representations are generated is proposed by introducing a feature averaging approach that combines encodings from multiple transformations of the original input.

Improving Transformation Invariance in Contrastive Representation Learning

A training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation is introduced, and a change to how test time representations are generated is proposed by introducing a feature averaging approach that combines encodings from multiple transformations of the original input.

F AIR N ORMALIZING F LOWS

This work presents Fair Normalizing Flows (FNF), a new approach offering more rigorous fairness guarantees for learned representations and experiments demonstrate the effectiveness of FNF in enforcing various group fairness notions, as well as other attractive properties such as interpretability and transfer learning.

Discovery and Separation of Features for Invariant Representation Learning

A framework for training robust neural networks that induces invariance to nuisances through learning to discover and separate predictive and nuisance factors of data is proposed.

Scalable Infomin Learning

This work proposes a new infomin learning approach, which uses a novel proxy metric to mutual information and derives an accurate and analytically computable approximation to this proxy metric, thereby removing the need of constructing neural network-based mutual information estimators.

Fair Normalizing Flows

This work presents Fair Normalizing Flows (FNF), a new approach offering more rigorous fairness guarantees for learned representations and experiments demonstrate the effectiveness of FNF in enforcing various group fairness notions, as well as other attractive properties such as interpretability and transfer learning.

De-STT: De-entaglement of unwanted Nuisances and Biases in Speech to Text System using Adversarial Forgetting

This paper studies the effect of de-entangling the accent information from the input speech signal on training STT systems using an information bottleneck architecture based on adversarial forgetting, which aims to enforce the model to learn general accent invariant speech representations.

Stabilizing Adversarial Invariance Induction from Divergence Minimization Perspective

This paper presents an analysis of the reasons for the optimization difficulties of AII, and provides a better optimization procedure by rethinking AII from a divergence minimization perspective, which consistently achieves near-optimal invariance in toy datasets with various configurations in which the original AII is catastrophically unstable.

Information-Theoretic Bias Assessment Of Learned Representations Of Pretrained Face Recognition

    Jiazhi LiWael AbdAlmageed
    Computer Science
    2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)
  • 2021
This work proposes an information-theoretic, independent bias assessment metric to identify degree of bias against protected demographic attributes from learned representations of pretrained facial recognition systems and establishes a benchmark metric.

36 References

Controllable Invariance through Adversarial Feature Learning

This paper shows that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance on three benchmark tasks.

Invariant Representations without Adversarial Training

It is shown that adversarial training is unnecessary and sometimes counter-productive; this work casts invariant representation learning as a single information-theoretic objective that can be directly optimized.

Unsupervised Adversarial Invariance

This work presents a novel unsupervised invariance induction framework for neural networks that learns a split representation of data through competitive training between the prediction task and a reconstruction task coupled with disentanglement, without needing any labeled information about nuisance factors or domain knowledge.

Unified Adversarial Invariance

This model outperforms previous works at inducing invariance to nuisance factors without using any labeled information about such variables, and achieves state-of-the-art performance at learning independence to biasing factors in fairness settings.

Deep Variational Information Bottleneck

It is shown that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

THE VARIATIONAL FAIR AUTOENCODER

This model is based on a variational autoencoding architecture with priors that encourage independence between sensitive and latent factors of variation with an additional penalty term based on the “Maximum Mean Discrepancy” (MMD) measure.

Information Dropout: Learning Optimal Representations Through Noisy Computation

It is proved that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.

Domain-Adversarial Training of Neural Networks

A new representation learning approach for domain adaptation, in which data at training and test time come from similar but different distributions, which can be achieved in almost any feed-forward model by augmenting it with few standard layers and a new gradient reversal layer.

Fader Networks: Manipulating Images by Sliding Attributes

A new encoder-decoder architecture that is trained to reconstruct images by disentangling the salient information of the image and the values of attributes directly in the latent space is introduced, which results in much simpler training schemes and nicely scales to multiple attributes.

Emergence of Invariance and Disentanglement in Deep Representations

It is shown that in a deep neural network invariance to nuisance factors is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations.