The Power of Depth for Feedforward Neural Networks

@inproceedings{Eldan2015ThePO,
  title={The Power of Depth for Feedforward Neural Networks},
  author={Ronen Eldan and Ohad Shamir},
  booktitle={COLT},
  year={2015}
}
We show that there is a simple (approximately radial) function on R d , expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all known activation functions, including rectified linear units, sigmoids and thresholds, and formally demonstrates that depth ‐ even if increased by 1 ‐ can be exponentially more valuable… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-10 OF 209 CITATIONS

A Depth Evidence Score Fusion Algorithm for Chinese Medical Intelligence Question Answering System

  • Journal of healthcare engineering
  • 2018
VIEW 5 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

Automated Architecture Design for Deep Neural Networks

Steven Abreu
  • ArXiv
  • 2019
VIEW 7 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

Deep Learning Volatility

Blanka Horvath, Aitor Muguruza, Mehdi Tomas
  • 2019
VIEW 4 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Efficient Deep Learning of GMMs

VIEW 5 EXCERPTS
CITES RESULTS & BACKGROUND
HIGHLY INFLUENCED

Limiting Network Size within Finite Bounds for Optimization

VIEW 4 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

Random ReLU Features: Universality, Approximation, and Composition

VIEW 6 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Universal Approximation with Quadratic Deep Networks

VIEW 9 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Deep Radial Kernel Networks: Approximating Radially Symmetric Functions with Deep Networks

VIEW 10 EXCERPTS
CITES RESULTS, METHODS & BACKGROUND
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2015
2019

CITATION STATISTICS

  • 17 Highly Influenced Citations

  • Averaged 61 Citations per year from 2017 through 2019