Accelerating deep neural network training with inconsistent stochastic gradient descent

@article{Wang2017AcceleratingDN,
  title={Accelerating deep neural network training with inconsistent stochastic gradient descent},
  author={Linnan Wang and Yi Yang and Martin Renqiang Min and Srimat T. Chakradhar},
  journal={Neural networks : the official journal of the International Neural Network Society},
  year={2017},
  volume={93},
  pages={219-229}
}
Stochastic Gradient Descent (SGD) updates Convolutional Neural Network (CNN) with a noisy gradient computed from a random batch, and each batch evenly updates the network once in an epoch. This model applies the same training effort to each batch, but it overlooks the fact that the gradient variance, induced by Sampling Bias and Intrinsic Image Difference… CONTINUE READING