Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets

@article{Heinecke2020RefinementAU,
  title={Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets},
  author={Andreas Heinecke and Jinn Ho and Wen-Liang Hwang},
  journal={IEEE Signal Processing Letters},
  year={2020},
  volume={27},
  pages={1175-1179}
}
We construct a highly regular and simple structured class of sparsely connected convolutional neural networks with rectifier activations that provide universal function approximation in a coarse-to-fine manner with increasing number of layers. The networks are localized in the sense that local changes in the function to be approximated only require local changes in the final layer of weights. At the core of the construction lies the fact that the characteristic function can be derived from a… 

Figures from this paper

Piecewise-Linear Activations or Analytic Activation Functions: Which Produce More Expressive Neural Networks?
TLDR
The main result demonstrates that deep networks with piecewise linear activation (e.g. ReLU or PReLU) are fundamentally more expressive than deep feedforward networks with analytic activation functions and is further explained by quantitatively demonstrating the “separation phenomenon” between the networks in NN ReLU + Pool.
Learning DNN networks using un-rectifying ReLU with compressed sensing application
TLDR
It is demonstrated that the optimal solution to a combinatorial optimization problem can be preserved by relaxing the discrete domains of activation variables to closed intervals, which makes it easier to learn a network using methods developed for real-domain constrained optimization.
Approximation with Neural Networks in Variable Lebesgue Spaces
TLDR
It is shown that, whenever the exponent function of the space is bounded, every function can be approximated with shallow neural networks with any desired accuracy, and the universality of the approximation depending on the boundedness of the exponentfunction is determined.
Handling Vanishing Gradient Problem Using Artificial Derivative
TLDR
This work proposed a method replacing the original derivative function with an artificial derivative in a pertinent way to effectively alleviate the vanishing gradient problem for both ReLU and sigmoid function with few computational cost.
Reliability on Deep Learning Models: A Comprehensive Observation
TLDR
The essential background and kernel techniques in deep learning, such as downsampling and nonlinear discontinuity, are introduced and the inherent structural flaws of deep learning and the risk of unreliability that can result from it are discussed.
Deep Neural Network-Based Detection and Partial Response Equalization for Multilayer Magnetic Recording
TLDR
This article uses MLMR waveforms generated using a grain switching probability (GSP) model that is trained on realistic micromagnetic simulations to propose three systems for equalization and detection and shows that the first system outperforms the traditional 2-D linear minimum mean squared error (2-D-LMMSE) equalizer.
Approximation of CIEDE2000 color closeness function using Neuro-Fuzzy networks
TLDR
The aim of this research was to develop an alternative to the unified closeness formula, which could be built on expert knowledge or be adapted to fit a particular target.
Deep Neural Network Media Noise Predictor Turbo-Detection System for 1-D and 2-D High-Density Magnetic Recording
TLDR
The presented BCJR-LDPC-CNN turbo-detection system obtains 3.877 Terabits per square inch (T/bin2) areal density for 11.4 Tg/in2 GFP model data, which is among the highest areal densities reported to date.
Interactive Deep Learning for Shelf Life Prediction of Muskmelons Based on an Active Learning Approach
TLDR
This paper proposes k-Determinantal Point Processes (k-DPP), which is a purely diversity-based method that allows to take influence on the exploration within the feature space based on the chosen subset k, and suggests the use of diversity- based acquisition when only a few labelled samples are available, allowing for better exploration.
Segmentation by Test-Time Optimization (TTO) for CBCT-based Adaptive Radiation Therapy
TLDR
The proposed test-time optimization (TTO) method is well-suited for online ART and can boost segmentation accuracy for DL-based DIR models, especially for outlier patients where the pre-trained models fail.
...
1
2
...

References

SHOWING 1-10 OF 25 REFERENCES
Optimal approximation of piecewise smooth functions using deep ReLU neural networks
Optimal Approximation with Sparsely Connected Deep Neural Networks
TLDR
All function classes that are optimally approximated by a general class of representation systems---so-called affine systems---can be approximating by deep neural networks with minimal connectivity and memory requirements, and it is proved that the lower bounds are achievable for a broad family of function classes.
Un-Rectifying Non-Linear Networks for Signal Representation
TLDR
A novel technique is proposed to “un-rectify” the nonlinear activations into data-dependent linear equations and constraints, from which explicit expressions for the affine linear operators, their domains and ranges in terms of the network parameters are derived.
Convolutional Rectifier Networks as Generalized Tensor Decompositions
TLDR
Developing effective methods for training convolutional arithmetic circuits may give rise to a deep learning architecture that is provably superior to Convolutional rectifier networks, which has so far been overlooked by practitioners.
Error bounds for approximations with deep ReLU networks
Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
TLDR
It is proved that ReLU nets with width d + 1 can approximate any continuous convex function of d variables arbitrarily well, and quantitative depth estimates for the rate of approximation of any continuous scalar function on the d-dimensional cube [ 0, 1 ] d by ReLUnets with widthd + 3 .
Understanding Deep Neural Networks with Rectified Linear Units
TLDR
The gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature, and a new lowerbound on the number of affine pieces is shown, larger than previous constructions in certain regimes of the network architecture.
Approximation capabilities of multilayer feedforward networks
  • K. Hornik
  • Computer Science
    Neural Networks
  • 1991
The Geometry of Deep Networks: Power Diagram Subdivision
TLDR
It is demonstrated that each MASO layer's input space partitioning corresponds to a {\em power diagram} (an extension of the classical Voronoi tiling) with a number of regions that grows exponentially with respect to the number of units (neurons).
The Power of Depth for Feedforward Neural Networks
TLDR
It is shown that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, unless its width is exponential in the dimension.
...
1
2
3
...