Flow of Renyi information in deep neural networks

@article{Huang2016FlowOR,
  title={Flow of Renyi information in deep neural networks},
  author={Che-Wei Huang and Shrikanth Narayanan},
  journal={2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP)},
  year={2016},
  pages={1-6}
}
We propose a rate-distortion based deep neural network (DNN) training algorithm using a smooth matrix functional on the manifold of positive semi-definite matrices as the non-parametric entropy estimator. The objective in the optimization function includes not only the measure of performance of the output layer but also the measure of information distortion between consecutive layers in order to produce a concise representation of its input on each layer. An experiment on speech emotion… CONTINUE READING