Optimal ensemble averaging of neural networks


Based on an observation about the different effect of ensemble averaging on the bias and variance portions of the prediction error, we discuss training methodologies for ensembles of networks. We demonstrate the effect of variance reduction and present a method of extrapolation to the limit of an infinite ensemble. A significant reduction of variance is obtained by averaging just over initial conditions of the neural networks, without varying architectures or training sets. The minimum of the ensemble prediction error is reached later than that of a single network. In the vicinity of the minimum, the ensemble prediction error appears to be flatter than that of the single network, thus simplifying optimal stopping decision. The results are demonstrated on sunspots data, where the predictions are among the best obtained, and on the 1993 energy prediction competition data set B.

11 Figures and Tables

Citations per Year

245 Citations

Semantic Scholar estimates that this publication has 245 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Naftaly1997OptimalEA, title={Optimal ensemble averaging of neural networks}, author={Ury Naftaly and Nathan Intrator and David Horn}, year={1997} }