Deep learning and the information bottleneck principle

@article{Tishby2015DeepLA,
  title={Deep learning and the information bottleneck principle},
  author={Naftali Tishby and Noga Zaslavsky},
  journal={2015 IEEE Information Theory Workshop (ITW)},
  year={2015},
  pages={1-5}
}
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite… CONTINUE READING