# Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

@article{Ciresan2010DeepBS, title={Deep, Big, Simple Neural Nets for Handwritten Digit Recognition}, author={Dan C. Ciresan and Ueli Meier and Luca Maria Gambardella and J{\"u}rgen Schmidhuber}, journal={Neural Computation}, year={2010}, volume={22}, pages={3207-3220} }

Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning.

#### Topics from this paper

#### 840 Citations

Deep Big Multilayer Perceptrons for Digit Recognition

- Computer Science
- Neural Networks: Tricks of the Trade
- 2012

All you need to achieve this until 2011 best result are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning. Expand

Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs

- Computer Science
- ArXiv
- 2011

Another substantial improvement is reported: 0.31% obtained using a committee of MLPs using simple but deep MLPs, outperforming all the previous more complex methods. Expand

Input Transformation and Output Combination for Improved Handwritten Digit Recognition

- Computer Science
- 2015

A specific set of input pattern transformations is presented that achieves good results with modestly sized Neural Networks using some heuristics for the construction of an ensemble allows reaching low error rates. Expand

Handwritten Digit Recognition with Pattern Transformations and Neural Network Averaging

- Computer Science
- ICANN
- 2013

A relatively modest sized Neural Network trained with standard Back Propagation and combined with a set of input pattern transformations is proposed, giving an encouraging error rate of 0.34% measured on the MNIST dataset. Expand

Fast Handwritten Digit Recognition with Multilayer Ensemble Extreme Learning Machine

- Computer Science
- SGAI Conf.
- 2015

A novel classifier based on Extreme Learning Machine is proposed that achieves competitive accuracy results while keeping training times low and is called multilayer ensemble Extreme learning Machine. Expand

Combining additive input noise annealing and pattern transformations for improved handwritten character recognition

- Computer Science
- Expert Syst. Appl.
- 2014

Using a modest sized ANN, the proposed combination of input data transformations enables the achievement of a test error as low as 0.43%, which is up to standard compared to other more complex neural architectures like Convolutional or Deep Neural Networks. Expand

Multi-column deep neural networks for image classification

- Computer Science
- 2012 IEEE Conference on Computer Vision and Pattern Recognition
- 2012

On the very competitive MNIST handwriting benchmark, this method is the first to achieve near-human performance and improves the state-of-the-art on a plethora of common image classification benchmarks. Expand

Digit Recognition Using Convolution Neural Network

- Computer Science
- ArXiv
- 2020

The main objective of this work is to obtain highest accuracy 99.15% by using convolution neural network (CNN) to recognize the digit without doing too much pre-processing of dataset. Expand

Better Digit Recognition with a Committee of Simple Neural Nets

- Computer Science
- 2011 International Conference on Document Analysis and Recognition
- 2011

A new method to train the members of a committee of one-hidden-layer neural nets is presented, which obtains a recognition error rate on the MNIST digit recognition benchmark set of 0.39%, on par with state-of-the-art recognition rates of more complicated systems. Expand

Efficient Handwritten Digit Recognition Using Normalized Cross-Correlation

- Mathematics
- 2016

The aim of handwriting recognition is to recognize series of handwritten characters by machines. It is a popular field in computer vision and has several commercial applications which requires… Expand

#### References

SHOWING 1-10 OF 34 REFERENCES

A trainable feature extractor for handwritten digit recognition

- Computer Science
- Pattern Recognit.
- 2007

A trainable feature extractor based on the LeNet5 convolutional neural network architecture is introduced to solve the first problem in a black box scheme without prior knowledge on the data and the results show that the system can outperform both SVMs and Le net5 while providing performances comparable to the best performance on this database. Expand

Accelerating Large-Scale Convolutional Neural Networks with Parallel Graphics Multiprocessors

- Computer Science
- ICANN
- 2010

This work has adapted the inherent multi-level parallelism of CNNs for Nvidia's CUDA GPU architecture to accelerate the training by two orders of magnitude, allowing to apply CNN architectures to pattern recognition tasks on datasets with high-resolution natural images. Expand

Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

- Mathematics, Computer Science
- AISTATS
- 2007

We show how to pretrain and fine-tune a multilayer neural network to learn a nonlinear transformation from the input space to a lowdimensional feature space in which K-nearest neighbour… Expand

Gradient-based learning applied to document recognition

- Computer Science
- 1998

This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques. Expand

High Performance Convolutional Neural Networks for Document Processing

- Computer Science
- 2006

Three novel approaches to speeding up CNNs are presented: a) unrolling convolution, b) using BLAS (basic linear algebra subroutines), and c) using GPUs (graphic processing units). Expand

Reducing the Dimensionality of Data with Neural Networks

- Computer Science, Medicine
- Science
- 2006

This work describes an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data. Expand

Training Invariant Support Vector Machines

- Computer Science
- Machine Learning
- 2004

This work reports the recent achievement of the lowest reported test error on the well-known MNIST digit recognition benchmark task, with SVM training times that are also significantly faster than previous SVM methods. Expand

Best practices for convolutional neural networks applied to visual document analysis

- Computer Science
- Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings.
- 2003

A set of concrete bestpractices that document analysis researchers can use to get good results with neural networks, including a simple "do-it-yourself" implementation of convolution with a flexible architecture suitable for many visual document problems. Expand

Using GPUs for machine learning algorithms

- Computer Science
- Eighth International Conference on Document Analysis and Recognition (ICDAR'05)
- 2005

This work proposes a generic 2-layer fully connected neural network GPU implementation which yields over 3/spl times/ speedup for both training and testing with respect to a 3 GHz P4 CPU. Expand

To recognize shapes, first learn to generate images.

- Computer Science, Medicine
- Progress in brain research
- 2007

This chapter describes several of the proposed algorithms and shows how they can be combined to produce hybrid methods that work efficiently in networks with many layers and millions of adaptive connections. Expand