Emergence of Invariance and Disentanglement in Deep Representations

@article{Achille2018EmergenceOI,
  title={Emergence of Invariance and Disentanglement in Deep Representations},
  author={Alessandro Achille and Stefano Soatto},
  journal={2018 Information Theory and Applications Workshop (ITA)},
  year={2018},
  pages={1-9}
}
Using established principles from Information Theory and Statistics, we show that in a deep neural network invariance to nuisance factors is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations. We then show that, in order to avoid memorization, we need to limit the quantity of information stored in the weights, which leads to a novel usage of the… CONTINUE READING
Highly Cited
This paper has 19 citations. REVIEW CITATIONS

Citations

Publications citing this paper.
Showing 1-10 of 14 extracted citations

References

Publications referenced by this paper.
Showing 1-10 of 35 references

Similar Papers

Loading similar papers…