Corpus ID: 211076199

On Predictive Information in RNNs.

  title={On Predictive Information in RNNs.},
  author={Zhe Dong and Deniz Oktay and Ben Poole and Alexander Amir Alemi},
  journal={arXiv: Learning},
  • Zhe Dong, Deniz Oktay, +1 author Alexander Amir Alemi
  • Published 2020
  • Mathematics, Computer Science
  • arXiv: Learning
  • Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. Empirically, we find that RNNs are suboptimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for… CONTINUE READING


    Predictive information in a sensory population
    • 152
    • Highly Influential
    • PDF
    Predictability , Complexity , and Learning
    • 335
    • Highly Influential
    • PDF
    On Variational Bounds of Mutual Information
    • 129
    • PDF
    Formal Limitations on the Measurement of Mutual Information
    • 50
    • PDF
    A Neural Representation of Sketch Drawings
    • 347
    • PDF
    Representation Learning with Contrastive Predictive Coding
    • 835
    • PDF
    Attention is All you Need
    • 14,957
    • PDF
    Deep Variational Information Bottleneck
    • 493
    • PDF