# If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks

@article{Pretorius2019IfDL, title={If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks}, author={Arnu Pretorius and Elan Van Biljon and B. Niekerk and Ryan Eloff and Matthew Reynard and S. James and Benjamin Rosman and H. Kamper and S. Kroon}, journal={ArXiv}, year={2019}, volume={abs/1910.05725} }

Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter… CONTINUE READING

One Citation

On the expected behaviour of noise regularised deep neural networks as Gaussian processes

- Computer Science, Mathematics
- 2019

- 2
- PDF

#### References

##### Publications referenced by this paper.

SHOWING 1-10 OF 33 REFERENCES

Understanding the difficulty of training deep feedforward neural networks

- Computer Science, Mathematics
- 2010

- 8,543
- Highly Influential
- PDF

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

- Computer Science
- 2015

- 8,034
- Highly Influential
- PDF

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

- Mathematics, Computer Science
- 2016

- 2,241
- PDF

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

- Computer Science, Physics
- 2014

- 919
- Highly Influential
- PDF

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

- Mathematics, Computer Science
- 2017

- 122
- PDF