ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs

@inproceedings{Gholami2019ANODEUA,
  title={ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs},
  author={Amir Gholami and K. Keutzer and G. Biros},
  booktitle={IJCAI},
  year={2019}
}
Residual neural networks can be viewed as the forward Euler discretization of an Ordinary Differential Equation (ODE) with a unit time step. [...] Key Method However, we will show that this approach may lead to several problems: (i) it may be numerically unstable for ReLU/non-ReLU activations and general convolution operators, and (ii) the proposed optimize-then-discretize approach may lead to divergent training due to inconsistent gradients for small time step sizes. We discuss the underlying problems, and to…Expand
ANODEV2: A Coupled Neural ODE Evolution Framework
ANODEV2: A Coupled Neural ODE Framework
SNODE: Spectral Discretization of Neural ODEs for System Identification
Meta-Solver for Neural Ordinary Differential Equations
Accelerating Neural ODEs with Spectral Elements
Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics
Deterministic Inference of Neural Stochastic Differential Equations
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 37 REFERENCES
Neural Ordinary Differential Equations
Deep Neural Networks Motivated by Partial Differential Equations
Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
Reversible Architectures for Arbitrarily Deep Residual Neural Networks
Stable Architectures for Deep Neural Networks
Training Deep Nets with Sublinear Memory Cost
A comparison of second-order methods for deep convolutional neural networks
Neural Networks: Tricks of the Trade
Memory-Efficient Backpropagation Through Time
...
1
2
3
4
...