Corpus ID: 202539613

LCA: Loss Change Allocation for Neural Network Training

  title={LCA: Loss Change Allocation for Neural Network Training},
  author={Janice Lan and Rosanne Liu and Hattie Zhou and J. Yosinski},
  • Janice Lan, Rosanne Liu, +1 author J. Yosinski
  • Published in NeurIPS 2019
  • Mathematics, Computer Science
  • Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is limited, with a single scalar loss being the most common viewport into this high-dimensional, dynamic process. We propose a new window into training called Loss Change Allocation (LCA), in which credit for changes to the network loss is conservatively partitioned to the parameters. This measurement is accomplished by… CONTINUE READING

    Figures, Tables, and Topics from this paper.

    The Break-Even Point on Optimization Trajectories of Deep Neural Networks
    • 12
    • PDF
    TRADI: Tracking deep neural network weight distributions
    • 3
    • PDF
    Partitioned integrators for thermodynamic parameterization of neural networks
    • 4
    • PDF
    SpaceNet: Make Free Space For Continual Learning
    Understanding Learning Dynamics for Neural Machine Translation
    Convolutional neural networks compression with low rank and sparse tensor decompositions
    Understanding Regularisation Methods for Continual Learning


    Publications referenced by this paper.
    Deep Residual Learning for Image Recognition
    • 50,343
    • Highly Influential
    • PDF
    On the importance of initialization and momentum in deep learning
    • 2,506
    • PDF
    On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
    • 994
    • PDF
    Opening the Black Box of Deep Neural Networks via Information
    • 587
    • PDF
    Optimization Methods for Large-Scale Machine Learning
    • 1,030
    • PDF
    On the Quality of the Initial Basin in Overspecified Neural Networks
    • 98
    • PDF