Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

@article{Ciresan2010DeepBS,
  title={Deep, Big, Simple Neural Nets for Handwritten Digit Recognition},
  author={Dan C. Ciresan and Ueli Meier and Luca Maria Gambardella and J{\"u}rgen Schmidhuber},
  journal={Neural Computation},
  year={2010},
  volume={22},
  pages={3207-3220}
}
Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning. 

Deep Big Multilayer Perceptrons for Digit Recognition

TLDR
All you need to achieve this until 2011 best result are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning.

Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs

TLDR
Another substantial improvement is reported: 0.31% obtained using a committee of MLPs using simple but deep MLPs, outperforming all the previous more complex methods.

Input Transformation and Output Combination for Improved Handwritten Digit Recognition

TLDR
A specific set of input pattern transformations is presented that achieves good results with modestly sized Neural Networks using some heuristics for the construction of an ensemble allows reaching low error rates.

Handwritten Digit Recognition with Pattern Transformations and Neural Network Averaging

TLDR
A relatively modest sized Neural Network trained with standard Back Propagation and combined with a set of input pattern transformations is proposed, giving an encouraging error rate of 0.34% measured on the MNIST dataset.

Fast Handwritten Digit Recognition with Multilayer Ensemble Extreme Learning Machine

TLDR
A novel classifier based on Extreme Learning Machine is proposed that achieves competitive accuracy results while keeping training times low and is called multilayer ensemble Extreme learning Machine.

Multi-column deep neural networks for image classification

TLDR
On the very competitive MNIST handwriting benchmark, this method is the first to achieve near-human performance and improves the state-of-the-art on a plethora of common image classification benchmarks.

Digit Recognition Using Convolution Neural Network

TLDR
The main objective of this work is to obtain highest accuracy 99.15% by using convolution neural network (CNN) to recognize the digit without doing too much pre-processing of dataset.

Better Digit Recognition with a Committee of Simple Neural Nets

TLDR
A new method to train the members of a committee of one-hidden-layer neural nets is presented, which obtains a recognition error rate on the MNIST digit recognition benchmark set of 0.39%, on par with state-of-the-art recognition rates of more complicated systems.

Efficient Handwritten Digit Recognition Using Normalized Cross-Correlation

TLDR
The applied research proposes a feasible statistical method and its comparison which showed competitive results in the evaluation phase based on prior works and used the famous MNIST dataset of handwritten digits for testing purposes.
...

References

SHOWING 1-10 OF 36 REFERENCES

A trainable feature extractor for handwritten digit recognition

Accelerating Large-Scale Convolutional Neural Networks with Parallel Graphics Multiprocessors

TLDR
This work has adapted the inherent multi-level parallelism of CNNs for Nvidia's CUDA GPU architecture to accelerate the training by two orders of magnitude, allowing to apply CNN architectures to pattern recognition tasks on datasets with high-resolution natural images.

Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

We show how to pretrain and fine-tune a multilayer neural network to learn a nonlinear transformation from the input space to a lowdimensional feature space in which K-nearest neighbour

Gradient-based learning applied to document recognition

TLDR
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.

High Performance Convolutional Neural Networks for Document Processing

TLDR
Three novel approaches to speeding up CNNs are presented: a) unrolling convolution, b) using BLAS (basic linear algebra subroutines), and c) using GPUs (graphic processing units).

Reducing the Dimensionality of Data with Neural Networks

TLDR
This work describes an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

Training Invariant Support Vector Machines

TLDR
This work reports the recent achievement of the lowest reported test error on the well-known MNIST digit recognition benchmark task, with SVM training times that are also significantly faster than previous SVM methods.

Best practices for convolutional neural networks applied to visual document analysis

TLDR
A set of concrete bestpractices that document analysis researchers can use to get good results with neural networks, including a simple "do-it-yourself" implementation of convolution with a flexible architecture suitable for many visual document problems.

Using GPUs for machine learning algorithms

TLDR
This work proposes a generic 2-layer fully connected neural network GPU implementation which yields over 3/spl times/ speedup for both training and testing with respect to a 3 GHz P4 CPU.

To recognize shapes, first learn to generate images.