# Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets

@article{Heinecke2020RefinementAU,
title={Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets},
author={Andreas Heinecke and Jinn Ho and Wen-Liang Hwang},
journal={IEEE Signal Processing Letters},
year={2020},
volume={27},
pages={1175-1179}
}
• Published 25 June 2020
• Computer Science
• IEEE Signal Processing Letters
We construct a highly regular and simple structured class of sparsely connected convolutional neural networks with rectifier activations that provide universal function approximation in a coarse-to-fine manner with increasing number of layers. The networks are localized in the sense that local changes in the function to be approximated only require local changes in the final layer of weights. At the core of the construction lies the fact that the characteristic function can be derived from a…
11 Citations

## Figures from this paper

Piecewise-Linear Activations or Analytic Activation Functions: Which Produce More Expressive Neural Networks?
• Computer Science
ArXiv
• 2022
The main result demonstrates that deep networks with piecewise linear activation (e.g. ReLU or PReLU) are fundamentally more expressive than deep feedforward networks with analytic activation functions and is further explained by quantitatively demonstrating the “separation phenomenon” between the networks in NN ReLU + Pool.
Learning DNN networks using un-rectifying ReLU with compressed sensing application
• Computer Science
ArXiv
• 2021
It is demonstrated that the optimal solution to a combinatorial optimization problem can be preserved by relaxing the discrete domains of activation variables to closed intervals, which makes it easier to learn a network using methods developed for real-domain constrained optimization.
Approximation with Neural Networks in Variable Lebesgue Spaces
• Mathematics, Computer Science
ArXiv
• 2020
It is shown that, whenever the exponent function of the space is bounded, every function can be approximated with shallow neural networks with any desired accuracy, and the universality of the approximation depending on the boundedness of the exponentfunction is determined.
Handling Vanishing Gradient Problem Using Artificial Derivative
• Computer Science
IEEE Access
• 2021
This work proposed a method replacing the original derivative function with an artificial derivative in a pertinent way to effectively alleviate the vanishing gradient problem for both ReLU and sigmoid function with few computational cost.
Reliability on Deep Learning Models: A Comprehensive Observation
• Computer Science
2020 6th International Symposium on System and Software Reliability (ISSSR)
• 2020
The essential background and kernel techniques in deep learning, such as downsampling and nonlinear discontinuity, are introduced and the inherent structural flaws of deep learning and the risk of unreliability that can result from it are discussed.
Deep Neural Network-Based Detection and Partial Response Equalization for Multilayer Magnetic Recording
• Computer Science
IEEE Transactions on Magnetics
• 2021
This article uses MLMR waveforms generated using a grain switching probability (GSP) model that is trained on realistic micromagnetic simulations to propose three systems for equalization and detection and shows that the first system outperforms the traditional 2-D linear minimum mean squared error (2-D-LMMSE) equalizer.
Approximation of CIEDE2000 color closeness function using Neuro-Fuzzy networks
• Computer Science
• 2021
The aim of this research was to develop an alternative to the unified closeness formula, which could be built on expert knowledge or be adapted to fit a particular target.
Deep Neural Network Media Noise Predictor Turbo-Detection System for 1-D and 2-D High-Density Magnetic Recording
• Computer Science
IEEE Transactions on Magnetics
• 2021
The presented BCJR-LDPC-CNN turbo-detection system obtains 3.877 Terabits per square inch (T/bin2) areal density for 11.4 Tg/in2 GFP model data, which is among the highest areal densities reported to date.
Interactive Deep Learning for Shelf Life Prediction of Muskmelons Based on an Active Learning Approach
• Computer Science
Sensors
• 2022
This paper proposes k-Determinantal Point Processes (k-DPP), which is a purely diversity-based method that allows to take influence on the exploration within the feature space based on the chosen subset k, and suggests the use of diversity- based acquisition when only a few labelled samples are available, allowing for better exploration.
• Computer Science
ArXiv
• 2022
The proposed test-time optimization (TTO) method is well-suited for online ART and can boost segmentation accuracy for DL-based DIR models, especially for outlier patients where the pre-trained models fail.

## References

SHOWING 1-10 OF 25 REFERENCES
Optimal Approximation with Sparsely Connected Deep Neural Networks
• Computer Science
SIAM J. Math. Data Sci.
• 2019
All function classes that are optimally approximated by a general class of representation systems---so-called affine systems---can be approximating by deep neural networks with minimal connectivity and memory requirements, and it is proved that the lower bounds are achievable for a broad family of function classes.
Un-Rectifying Non-Linear Networks for Signal Representation
• Computer Science
IEEE Transactions on Signal Processing
• 2020
A novel technique is proposed to “un-rectify” the nonlinear activations into data-dependent linear equations and constraints, from which explicit expressions for the affine linear operators, their domains and ranges in terms of the network parameters are derived.
Convolutional Rectifier Networks as Generalized Tensor Decompositions
• Computer Science
ICML
• 2016
Developing effective methods for training convolutional arithmetic circuits may give rise to a deep learning architecture that is provably superior to Convolutional rectifier networks, which has so far been overlooked by practitioners.
Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
It is proved that ReLU nets with width d + 1 can approximate any continuous convex function of d variables arbitrarily well, and quantitative depth estimates for the rate of approximation of any continuous scalar function on the d-dimensional cube [ 0, 1 ] d by ReLUnets with widthd + 3 .
Understanding Deep Neural Networks with Rectified Linear Units
• Computer Science, Mathematics
Electron. Colloquium Comput. Complex.
• 2017
The gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature, and a new lowerbound on the number of affine pieces is shown, larger than previous constructions in certain regimes of the network architecture.
The Geometry of Deep Networks: Power Diagram Subdivision
• Computer Science
NeurIPS
• 2019
It is demonstrated that each MASO layer's input space partitioning corresponds to a {\em power diagram} (an extension of the classical Voronoi tiling) with a number of regions that grows exponentially with respect to the number of units (neurons).
The Power of Depth for Feedforward Neural Networks
• Computer Science
COLT
• 2016
It is shown that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, unless its width is exponential in the dimension.