# Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets

@article{Heinecke2020RefinementAU, title={Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets}, author={Andreas Heinecke and Jinn Ho and Wen-Liang Hwang}, journal={IEEE Signal Processing Letters}, year={2020}, volume={27}, pages={1175-1179} }

We construct a highly regular and simple structured class of sparsely connected convolutional neural networks with rectifier activations that provide universal function approximation in a coarse-to-fine manner with increasing number of layers. The networks are localized in the sense that local changes in the function to be approximated only require local changes in the final layer of weights. At the core of the construction lies the fact that the characteristic function can be derived from a…

## 11 Citations

Piecewise-Linear Activations or Analytic Activation Functions: Which Produce More Expressive Neural Networks?

- Computer ScienceArXiv
- 2022

The main result demonstrates that deep networks with piecewise linear activation (e.g. ReLU or PReLU) are fundamentally more expressive than deep feedforward networks with analytic activation functions and is further explained by quantitatively demonstrating the “separation phenomenon” between the networks in NN ReLU + Pool.

Learning DNN networks using un-rectifying ReLU with compressed sensing application

- Computer ScienceArXiv
- 2021

It is demonstrated that the optimal solution to a combinatorial optimization problem can be preserved by relaxing the discrete domains of activation variables to closed intervals, which makes it easier to learn a network using methods developed for real-domain constrained optimization.

Approximation with Neural Networks in Variable Lebesgue Spaces

- Mathematics, Computer ScienceArXiv
- 2020

It is shown that, whenever the exponent function of the space is bounded, every function can be approximated with shallow neural networks with any desired accuracy, and the universality of the approximation depending on the boundedness of the exponentfunction is determined.

Handling Vanishing Gradient Problem Using Artificial Derivative

- Computer ScienceIEEE Access
- 2021

This work proposed a method replacing the original derivative function with an artificial derivative in a pertinent way to effectively alleviate the vanishing gradient problem for both ReLU and sigmoid function with few computational cost.

Reliability on Deep Learning Models: A Comprehensive Observation

- Computer Science2020 6th International Symposium on System and Software Reliability (ISSSR)
- 2020

The essential background and kernel techniques in deep learning, such as downsampling and nonlinear discontinuity, are introduced and the inherent structural flaws of deep learning and the risk of unreliability that can result from it are discussed.

Deep Neural Network-Based Detection and Partial Response Equalization for Multilayer Magnetic Recording

- Computer ScienceIEEE Transactions on Magnetics
- 2021

This article uses MLMR waveforms generated using a grain switching probability (GSP) model that is trained on realistic micromagnetic simulations to propose three systems for equalization and detection and shows that the first system outperforms the traditional 2-D linear minimum mean squared error (2-D-LMMSE) equalizer.

Approximation of CIEDE2000 color closeness function using Neuro-Fuzzy networks

- Computer Science
- 2021

The aim of this research was to develop an alternative to the unified closeness formula, which could be built on expert knowledge or be adapted to fit a particular target.

Deep Neural Network Media Noise Predictor Turbo-Detection System for 1-D and 2-D High-Density Magnetic Recording

- Computer ScienceIEEE Transactions on Magnetics
- 2021

The presented BCJR-LDPC-CNN turbo-detection system obtains 3.877 Terabits per square inch (T/bin2) areal density for 11.4 Tg/in2 GFP model data, which is among the highest areal densities reported to date.

Interactive Deep Learning for Shelf Life Prediction of Muskmelons Based on an Active Learning Approach

- Computer ScienceSensors
- 2022

This paper proposes k-Determinantal Point Processes (k-DPP), which is a purely diversity-based method that allows to take influence on the exploration within the feature space based on the chosen subset k, and suggests the use of diversity- based acquisition when only a few labelled samples are available, allowing for better exploration.

Segmentation by Test-Time Optimization (TTO) for CBCT-based Adaptive Radiation Therapy

- Computer ScienceArXiv
- 2022

The proposed test-time optimization (TTO) method is well-suited for online ART and can boost segmentation accuracy for DL-based DIR models, especially for outlier patients where the pre-trained models fail.

## References

SHOWING 1-10 OF 25 REFERENCES

Optimal approximation of piecewise smooth functions using deep ReLU neural networks

- Computer ScienceNeural Networks
- 2018

Optimal Approximation with Sparsely Connected Deep Neural Networks

- Computer ScienceSIAM J. Math. Data Sci.
- 2019

All function classes that are optimally approximated by a general class of representation systems---so-called affine systems---can be approximating by deep neural networks with minimal connectivity and memory requirements, and it is proved that the lower bounds are achievable for a broad family of function classes.

Un-Rectifying Non-Linear Networks for Signal Representation

- Computer ScienceIEEE Transactions on Signal Processing
- 2020

A novel technique is proposed to “un-rectify” the nonlinear activations into data-dependent linear equations and constraints, from which explicit expressions for the affine linear operators, their domains and ranges in terms of the network parameters are derived.

Convolutional Rectifier Networks as Generalized Tensor Decompositions

- Computer ScienceICML
- 2016

Developing effective methods for training convolutional arithmetic circuits may give rise to a deep learning architecture that is provably superior to Convolutional rectifier networks, which has so far been overlooked by practitioners.

Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

- Computer ScienceArXiv
- 2017

It is proved that ReLU nets with width d + 1 can approximate any continuous convex function of d variables arbitrarily well, and quantitative depth estimates for the rate of approximation of any continuous scalar function on the d-dimensional cube [ 0, 1 ] d by ReLUnets with widthd + 3 .

Understanding Deep Neural Networks with Rectified Linear Units

- Computer Science, MathematicsElectron. Colloquium Comput. Complex.
- 2017

The gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature, and a new lowerbound on the number of affine pieces is shown, larger than previous constructions in certain regimes of the network architecture.

Approximation capabilities of multilayer feedforward networks

- Computer ScienceNeural Networks
- 1991

The Geometry of Deep Networks: Power Diagram Subdivision

- Computer ScienceNeurIPS
- 2019

It is demonstrated that each MASO layer's input space partitioning corresponds to a {\em power diagram} (an extension of the classical Voronoi tiling) with a number of regions that grows exponentially with respect to the number of units (neurons).

The Power of Depth for Feedforward Neural Networks

- Computer ScienceCOLT
- 2016

It is shown that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, unless its width is exponential in the dimension.