Emergence of Invariance and Disentanglement in Deep Representations

  title={Emergence of Invariance and Disentanglement in Deep Representations},
  author={Alessandro Achille and Stefano Soatto},
  journal={2018 Information Theory and Applications Workshop (ITA)},
Using established principles from Information Theory and Statistics, we show that in a deep neural network invariance to nuisance factors is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations. We then show that, in order to avoid memorization, we need to limit the quantity of information stored in the weights, which leads to a novel usage of the… CONTINUE READING
Highly Cited
This paper has 19 citations. REVIEW CITATIONS


Publications citing this paper.
Showing 1-10 of 14 extracted citations


Publications referenced by this paper.
Showing 1-10 of 35 references

Similar Papers

Loading similar papers…