An Information Theoretic Interpretation to Deep Neural Networks

@article{Huang2019AnIT,
  title={An Information Theoretic Interpretation to Deep Neural Networks},
  author={Shao-Lun Huang and Xiangxiang Xu and Lizhong Zheng and Gregory W. Wornell},
  journal={2019 IEEE International Symposium on Information Theory (ISIT)},
  year={2019},
  pages={1984-1988}
}
It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the "universal feature selection" problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-2 OF 2 CITATIONS

Massive Access for 5G and Beyond

VIEW 1 EXCERPT
CITES BACKGROUND

On The Sample Complexity of HGR Maximal Correlation Functions

VIEW 1 EXCERPT
CITES BACKGROUND

References

Publications referenced by this paper.
SHOWING 1-10 OF 24 REFERENCES

and G

  • S.-L. Huang, A. Makur, L. Zheng
  • W. Wornell, “On universal features for high-dimensional learning and inference,” submitted to IEEE Trans. Inform. Theory
  • 2019
VIEW 6 EXCERPTS
HIGHLY INFLUENTIAL

Inception-v4

  • C. Szegedy, S. Ioffe, V. Vanhoucke, A. A. Alemi
  • inception-resnet and the impact of residual connections on learning.” in AAAI
  • 2017

and H

  • A. G. Howard, M. Zhu, +4 authors M. Andreetto
  • Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861
  • 2017

and L

  • M. Arjovsky, S. Chintala
  • Bottou, “Wasserstein gan,” arXiv preprint arXiv:1701.07875
  • 2017
VIEW 2 EXCERPTS

and L

  • G. Huang, Z. Liu, K. Q. Weinberger
  • van der Maaten, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, vol. 1, no. 2
  • 2017

Xception: Deep Learning with Depthwise Separable Convolutions

  • François Chollet
  • Computer Science
  • 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2016

and Z

  • C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens
  • Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
  • 2016