Corpus ID: 235694562

On the Expected Complexity of Maxout Networks

@article{Tseran2021OnTE,
  title={On the Expected Complexity of Maxout Networks},
  author={Hanna Tseran and Guido Mont{\'u}far},
  journal={ArXiv},
  year={2021},
  volume={abs/2107.00379}
}
Learning with neural networks relies on the complexity of the representable functions, but more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and… Expand

References

SHOWING 1-10 OF 49 REFERENCES
Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
TLDR
Instead of just counting the number of the linear regions, this paper studies their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. Expand
Complexity of Linear Regions in Deep Networks
TLDR
The theory suggests that, even after training, the number of linear regions is far below exponential, an intuition that matches the empirical observations and concludes that the practical expressivity of neural networks is likely far below that of the theoretical maximum, and this gap can be quantified. Expand
On the Number of Linear Regions of Deep Neural Networks
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. DeepExpand
Bounding and Counting Linear Regions of Deep Neural Networks
TLDR
The results indicate that a deep rectifier network can only have more linear regions than every shallow counterpart with same number of neurons if that number exceeds the dimension of the input. Expand
Deep ReLU Networks Have Surprisingly Few Activation Patterns
TLDR
This work shows empirically that the average number of activation patterns for ReLU networks at initialization is bounded by the total number of neurons raised to the input dimension, and suggests that realizing the full expressivity of deep networks may not be possible in practice, at least with current methods. Expand
Maxout Networks
TLDR
A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. Expand
A General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regions
TLDR
This work proposes ageneral computational approach to compute a tight upper bound of regions number for theoretically any network structures (e.g. DNN with all kind of skip connec-tions and residual structures). Expand
Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
TLDR
Tight bounds on the smallest eigenvalue of NTK matrices for deep ReLU nets are provided, both in the limiting case of infinite widths and for finite widths. Expand
Understanding the difficulty of training deep feedforward neural networks
TLDR
The objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks, to better understand these recent relative successes and help design better algorithms in the future. Expand
A Framework for the Construction of Upper Bounds on the Number of Affine Linear Regions of ReLU Feed-Forward Neural Networks
TLDR
By using explicit formulas for a Jordan-like decomposition of the involved matrices, the framework to derive upper bounds on the number of regions that feed-forward neural networks with ReLU activation functions are affine linear on is presented. Expand
...
1
2
3
4
5
...