Universality of Deep Convolutional Neural Networks

@article{Zhou2020UniversalityOD,
  title={Universality of Deep Convolutional Neural Networks},
  author={Ding-Xuan Zhou},
  journal={ArXiv},
  year={2020},
  volume={abs/1805.10769}
}
Convergence of Deep Convolutional Neural Networks
TLDR
It turns out the convergence of deep neural networks as the depth of the networks tends to infinity reduces to convergence of infinite products of matrices with increasing sizes, which has not been considered in the literature.
Optimal Learning Rates of Deep Convolutional Neural Networks: Additive Ridge Functions
TLDR
It is shown that, for additive ridge functions, convolutional neural networks followed by one fully connected layer with ReLU activation functions can reach optimal mini-max rates (up to a log factor) and the convergence rates are dimension independent.
Theory of Deep Convolutional Neural Networks II: Spherical Analysis
Convergence of Deep Neural Networks with General Activation Functions and Pooling
TLDR
This work studies the convergence of deep neural networks as the depth tends to infinity for two other important activation functions: the leaky ReLU and the sigmoid function, and establishes a weaker sufficient condition for uniform convergence ofDeep neural networks.
Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks
TLDR
Butterfly-net, a low-complexity CNN with structured and sparse across-channel connections, which aims at an optimal hierarchical function representation of the input signal, outperforms the hard-coded Butterfly-net and achieves similar accuracy as the trained CNN but with much less parameters.
Universal Consistency of Deep Convolutional Neural Networks
TLDR
It is proved that implementing empirical risk minimization on DCNNs with expansive convolution (with zero-padding) is strongly universally consistent.
Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions
Convergence Analysis of Deep Residual Networks
TLDR
A matrix-vector description of general deep neural networks with shortcut connections is given and an explicit expression for the networks is formulated by using the notions of activation domains and activation matrices to characterize the convergence of deep Residual Networks.
A Sparse Coding Interpretation of Neural Networks and Theoretical Implications
TLDR
A sparse coding interpretation of neural networks that have ReLU activation and of convolutional neural networks in particular is proposed and potentially more robust forward transformations are motivated by maintaining sparse priors in convolutionAL neural networks as well performing a stronger nonlinear transformation.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 33 REFERENCES
Deep distributed convolutional neural networks: Universality
TLDR
It is shown that these deep neural networks have the same order of computational complexity as the deep convolutional neural networks, and it is proved their universality of approximation.
ImageNet classification with deep convolutional neural networks
TLDR
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Deep vs. shallow networks : An approximation theory perspective
TLDR
A new definition of relative dimension is proposed to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.
Optimal Approximation with Sparsely Connected Deep Neural Networks
TLDR
All function classes that are optimally approximated by a general class of representation systems---so-called affine systems---can be approximating by deep neural networks with minimal connectivity and memory requirements, and it is proved that the lower bounds are achievable for a broad family of function classes.
Provable approximation properties for deep neural networks
Optimal approximation of piecewise smooth functions using deep ReLU neural networks
A Fast Learning Algorithm for Deep Belief Nets
TLDR
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
The Power of Depth for Feedforward Neural Networks
TLDR
It is shown that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, unless its width is exponential in the dimension.
Deep Learning
TLDR
Deep learning is making major advances in solving problems that have resisted the best attempts of the artificial intelligence community for many years, and will have many more successes in the near future because it requires very little engineering by hand and can easily take advantage of increases in the amount of available computation and data.
Learning with Hierarchical Gaussian Kernels
TLDR
It is shown that Gaussian kernels are universal and that SVMs using these kernels are universally consistent, and a parameter optimization method for the kernel parameters is described that is empirically compared to SVMs, random forests, a multiple kernel learning approach, and to some deep neural networks.
...
1
2
3
4
...