Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

@article{He2015DelvingDI,
  title={Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification},
  author={Kaiming He and X. Zhang and Shaoqing Ren and Jian Sun},
  journal={2015 IEEE International Conference on Computer Vision (ICCV)},
  year={2015},
  pages={1026-1034}
}
  • Kaiming He, X. Zhang, +1 author Jian Sun
  • Published 6 February 2015
  • Computer Science
  • 2015 IEEE International Conference on Computer Vision (ICCV)
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. [...] Key Method PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on the learnable activation and advanced…Expand
A Deep Convolutional Neural Network with Selection Units for Super-Resolution
  • Jae-Seok Choi, Munchurl Kim
  • Computer Science
    2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
  • 2017
TLDR
The proposed deep network with SUs, called SelNet, was top-5th ranked in NTIRE2017 Challenge, which has a much lower computation complexity compared to the top-4 entries, and experiment results show that the proposed SelNet outperforms the authors' baseline only with ReLU, and other state-of-the-art deep-learning-based SR methods.
Empirical Evaluation of Rectified Activations in Convolutional Network
TLDR
The experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results, and are negative on the common belief that sparsity is the key of good performance in ReLU.
FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks
  • Suo Qiu, Bolun Cai
  • Computer Science
    2018 24th International Conference on Pattern Recognition (ICPR)
  • 2018
TLDR
Experimental results show that FReLU achieves fast convergence and competitive performance on both plain and residual networks, and is designed to be simple and effective without exponential functions to maintain low-cost computation.
Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
  • V. Pandey
  • Computer Science, Mathematics
    ArXiv
  • 2020
TLDR
Thresholded exponential rectified linear unit (TERELU) activation function that works better in alleviating in overfitting: large weight update problem, and also gives good amount of non-linearity as compared to other linear rectifiers.
Beyond ImageNet: Deep Learning in Industrial Practice
TLDR
This chapter focuses on convolutional neural networks, which have since the seminal work of Krizhevsky et al. revolutionized image classification and started surpassing human performance on some benchmark data sets and can be successfully applied to other areas and problems with some local structure in the data.
Deep neural networks with Elastic Rectified Linear Units for object recognition
TLDR
A novel Elastic Rectified Linear Unit (EReLU) that focuses on processing the positive part of input is proposed that improves model fitting with no extra parameters and little overfitting risk and is able to further improve the performance of networks.
Deep Transfer Learning for Art Classification Problems
TLDR
This paper shows how DCNNs, which have been fine tuned on a large artistic collection, outperform the same architectures which are pre-trained on the ImageNet dataset only, when it comes to the classification of heritage objects from a different dataset.
Shallow and wide fractional max-pooling network for image classification
TLDR
This work empirically investigates the architecture of popular ConvNet models and tries to widen the network enough in the fixed depth to achieve a similar or better performance.
PokeBNN: A Binary Pursuit of Lightweight Accuracy
TLDR
PokeConv is proposed, a binary convolution block which improves quality of BNNs by techniques such as adding multiple residual paths, and tuning the activation function, and is applied to ResNet-50 and optimize ResNet’s initial convolutional layer which is hard to binarize.
DPReLU: Dynamic Parametric Rectified Linear Unit
TLDR
DPReLU is proposed that can control the overall functional shape of ReLU with four learnable parameters and is based on the ideas of the Parametric ReLU (PReLU) and Flexible Re LU (FReLU).
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 45 REFERENCES
Deeply-Supervised Nets
TLDR
The proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent, and extends techniques from stochastic gradient methods to analyze the algorithm.
ImageNet classification with deep convolutional neural networks
TLDR
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Convolutional neural networks at constrained time cost
  • Kaiming He, Jian Sun
  • Computer Science
    2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2015
TLDR
This paper investigates the accuracy of CNNs under constrained time cost, and presents an architecture that achieves very competitive accuracy in the ImageNet dataset, yet is 20% faster than “AlexNet” [14] (16.0% top-5 error, 10-view test).
Return of the Devil in the Details: Delving Deep into Convolutional Nets
TLDR
It is shown that the data augmentation techniques commonly applied to CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost, and it is identified that the dimensionality of the CNN output layer can be reduced significantly without having an adverse effect on performance.
Going deeper with convolutions
We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition
Very Deep Convolutional Networks for Large-Scale Image Recognition
TLDR
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
TLDR
Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
TLDR
This work revisits both the alignment step and the representation step by employing explicit 3D face modeling in order to apply a piecewise affine transformation, and derive a face representation from a nine-layer deep neural network.
Some Improvements on Deep Convolutional Neural Network Based Image Classification
TLDR
This paper summarizes the entry in the Imagenet Large Scale Visual Recognition Challenge 2013, which achieved a top 5 classification error rate and achieved over a 20% relative improvement on the previous year's winner.
On rectified linear units for speech processing
TLDR
This work shows that it can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units.
...
1
2
3
4
5
...