Learning capability and storage capacity of two-hidden-layer feedforward networks

@article{Huang2003LearningCA,
  title={Learning capability and storage capacity of two-hidden-layer feedforward networks},
  author={Guangbin Huang},
  journal={IEEE transactions on neural networks},
  year={2003},
  volume={14 2},
  pages={
          274-81
        }
}
  • G. Huang
  • Published 1 March 2003
  • Computer Science
  • IEEE transactions on neural networks
The problem of the necessary complexity of neural networks is of interest in applications. In this paper, learning capability and storage capacity of feedforward neural networks are considered. We markedly improve the recent results by introducing neural-network modularity logically. This paper rigorously proves in a constructive method that two-hidden-layer feedforward networks (TLFNs) with 2/spl radic/(m+2)N (/spl Lt/N) hidden neurons can learn any N distinct samples (x/sub i/, t/sub i/) with… 
View on PubMed
www3.ntu.edu.sg
659 Citations
Highly Influential Citations
31
Background Citations
267
Methods Citations
81
Results Citations
6

Figures from this paper

659 Citations

Simplification of a specific two-hidden-layer feedforward networks
  • Lihui Chen, G. Huang, C. Siew
  • Computer Science
    Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint
  • 2003
TLDR
A method is introduced to simplify the structure of the TLFNs by introducing a new type of quantizers that unite two previous neurons A/sup (p)/ and B/Sup (p/ into a single neuron.
A Real-Time Learning Algorithm for Two-Hidden-Layer Feedforward Networks
  • Q. Zhu, G. Huang, C. Siew
  • Computer Science
    2003 4th International Conference on Control and Automation Proceedings
  • 2003
TLDR
An improved constructive method of TLFN with real-time learning capacity is introduced to prove that both the training and generalization errors of the new TLFN can reach arbitrarily small values if sufficient distinctive training samples are provided.
Accelerated Optimal Topology Search for Two-Hidden-Layer Feedforward Neural Networks
TLDR
Two-hidden-layer feedforward neural networks are investigated for the existence of an optimal hidden node ratio and the heuristic n_{1} = int(0.5n_{h} + 1) reduced the complexity of an exhaustive search from quadratic, to linear in n, with very little penalty.
Upper bounds on the node numbers of hidden layers in MLPs
TLDR
An upper bound on the node number of each hidden layer for the most general feedforward neural networks called multilayer perceptrons (MLP), from an algebraic point of view is given.
Pulling back error to the hidden-node parameter technology: Single-hidden-layer feedforward network without output weight
TLDR
This paper indicates that in order to let SLFNs work as universal approximators, one may simply calculate the hidden node parameter only and the output weight is not needed at all and this proposed neural network architecture can be considered as a standard SLFN with fixing output weight equal to an unit vector.
On Theoretical Analysis of Single Hidden Layer Feedforward Neural Networks with Relu Activations
  • Guorui Shen, Ye Yuan
  • Computer Science
    2019 34rd Youth Academic Annual Conference of Chinese Association of Automation (YAC)
  • 2019
TLDR
This note considers extreme learning machine that adopts non-smooth function as activation, proposing that a Relu activated single hidden layer feedforward neural network (SLFN) is capable of fitting given training data points with zero error under the condition that sufficient hidden neurons are provided at the hidden layer.
On the Optimal Node Ratio between Hidden Layers: A Probabilistic Study
TLDR
The findings were that the heuristic n1 = 0.5nh + 1 has an average probability of at least 0.85 of finding a network with a generalisation error within 0.18% of the best generaliser.
APPROACH TO THE SYNTHESIS OF NEURAL NETWORK STRUCTURE DURING CLASSIFICATION
TLDR
It is shown that artificial neural network with a two hidden layer feed forward Neural network with d inputs, d neurons in the first hidden layer, 2d+2 neuron in the second hidden layer and with a sigmoidal infinitely differentiable function can solve classification and pattern problems with arbitrary accuracy.
An Overview of Sequential Learning Algorithms for Single Hidden Layer Networks: Current Issues & Future Trends
TLDR
A brief survey of the commonly used sequential-learning algorithms used with single hidden layer feed-forward neural networks is presented, and a glimpse at the different kinds that are available in the literature up until now, how they have developed throughout the years, and their relative execution is summarized.
Universal approximation using incremental constructive feedforward networks with random hidden nodes
TLDR
This paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer.
...
...

References

SHOWING 1-10 OF 32 REFERENCES
Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions
TLDR
This paper rigorously proves that standard single-hidden layer feedforward networks with at most N hidden neurons and with any bounded nonlinear activation function which has a limit at one infinity can learn N distinct samples with zero error.
Capabilities of a four-layered feedforward neural network: four layers versus three
TLDR
A proof is given showing that a three-layered feedforward network with N-1 hidden units can give any N input-target relations exactly, and a four-layering network is constructed and is found to give anyN input- target relations with a negligibly small error using only (N/2)+3 hidden units.
Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions
TLDR
Multilayer feedforward networks possess universal approximation capabilities by virtue of the presence of intermediate layers with sufficiently many parallel processors; the properties of the intermediate-layer activation function are not so crucial.
On the capabilities of multilayer perceptrons
  • E. Baum
  • Computer Science
    J. Complex.
  • 1988
UNIQUENESS OF WEIGHTS FOR NEURAL NETWORKS
TLDR
This paper assumes that not only satises (*), but also the following extra condition, which appeared above in the context of single-hidden layer nets with no osets: (k) (0) 6 = 0 for innitely many integers k.
Sample sizes for sigmoidal neural networks
This paper applies the theory of Probably Approximately Correct (PAC) learning to feedforward neural networks with sigmoidal activation functions. Despite the best known upper bound on the VC…
The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network
TLDR
Results in this paper show that if a large neural network is used for a pattern classification problem and the learning algorithm finds a network with small weights that has small squared error on the training patterns, then the generalization performance depends on the size of the weights rather than the number of weights.
Comments on "Approximation capability in C(Rn) by multilayer feedforward networks and related problems"
TLDR
The conjecture that the boundedness of the sigmoidal function is a necessary and sufficient condition for the validity of the approximation theorem is not correct and boundedness and unequal limits at infinities conditions on the activation functions are sufficient, but not necessary in C(Rn).
Approximation capability in C(R¯n) by multilayer feedforward networks and related problems
TLDR
It is found that the boundedness condition on the sigmoidal function plays an essential role in the approximation, as contrast to continuity or monotonity condition.
For Valid Generalization the Size of the Weights is More Important than the Size of the Network
This paper shows that if a large neural network is used for a pattern classification problem, and the learning algorithm finds a network with small weights that has small squared error on the…
...
...