An Information Theoretic Interpretation to Deep Neural Networks

@article{Huang2019AnIT,
  title={An Information Theoretic Interpretation to Deep Neural Networks},
  author={Shao-Lun Huang and Xiangxiang Xu and L. Zheng and G. Wornell},
  journal={2019 IEEE International Symposium on Information Theory (ISIT)},
  year={2019},
  pages={1984-1988}
}
It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the "universal feature selection" problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network… Expand

Paper Mentions

Blog Post
An Information-Theoretic Approach to Unsupervised Feature Selection for High-Dimensional Data
Slope and generalization properties of neural networks
An information-theoretic approach to unsupervised feature selection for high-dimensional data
Maximal Correlation Regression
On The Sample Complexity of HGR Maximal Correlation Functions
A Local Characterization for Wyner Common Information
On the Sample Complexity of HGR Maximal Correlation Functions for Large Datasets
Person Recognition with HGR Maximal Correlation on Multimodal Data
  • Yihua Liang, Fei Ma, Yang Li, Shao-Lun Huang
  • Computer Science
  • 2020 25th International Conference on Pattern Recognition (ICPR)
  • 2021
...
1
2
...

References

SHOWING 1-10 OF 33 REFERENCES
Deep learning and the information bottleneck principle
Understanding intermediate layers using linear classifier probes
Densely Connected Convolutional Networks
Xception: Deep Learning with Depthwise Separable Convolutions
  • François Chollet
  • Computer Science, Mathematics
  • 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2017
Wasserstein GAN
Rethinking the Inception Architecture for Computer Vision
...
1
2
3
4
...