• Corpus ID: 166228026

ODE$^2$VAE: Deep generative second order ODEs with Bayesian neural networks

@article{Yildiz2019ODE2VAEDG,
  title={ODE\$^2\$VAE: Deep generative second order ODEs with Bayesian neural networks},
  author={Çagatay Yildiz and Markus Heinonen and Harri L{\"a}hdesm{\"a}ki},
  journal={ArXiv},
  year={2019},
  volume={abs/1905.10994}
}
We present Ordinary Differential Equation Variational Auto-Encoder (ODE$^2$VAE), a latent second order ODE model for high-dimensional sequential data. Leveraging the advances in deep generative models, ODE$^2$VAE can simultaneously learn the embedding of high dimensional trajectories and infer arbitrarily complex continuous-time latent dynamics. Our model explicitly decomposes the latent space into momentum and position components and solves a second order ODE system, which is in contrast to… 

Figures and Tables from this paper

Heavy Ball Neural Ordinary Differential Equations
TLDR
This work proposes heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference, and verifies the advantages of HBNodes over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling.
Segmenting Hybrid Trajectories using Latent ODEs
TLDR
The Latent Segmented ODE (LatSegODE), which uses Latent ODEs to perform reconstruction and changepoint detection within hybrid trajectories featuring jump discontinuities and switching dynamical modes, and proposes usage of the marginal likelihood as a score function for PELT, circumventing the need for modelcomplexity-based penalization.
Dissecting Neural ODEs
TLDR
This work "open the box" and offers a system-theoretic perspective, including state augmentation strategies and robustness, with the aim of clarifying the influence of several design choices on the underlying dynamics.
Stochasticity in Neural ODEs: An Empirical Study
TLDR
It is demonstrated that neural SDE is able to outperform its deterministic counterpart, and data augmentation during the training improves the performance of both deterministic and stochastic versions of the same model.
Stochastic Latent Residual Video Prediction
TLDR
This paper introduces a novel stochastic temporal model whose dynamics are governed in a latent space by a residual update rule and naturally models video dynamics as it allows this simpler, more interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.
Continuous-Time Video Generation via Learning Motion Dynamics with Neural ODE
TLDR
A novel video generation approach that learns separate distributions for motion and appearance, the former modeled by neural Ordinary Differential Equation to learn natural motion dynamics, which not only quantitatively outperforms recent baselines for video generation, but also demonstrates versatile functionality such as dynamic frame rate manipulation and motion transfer between two datasets.
Neural ODE Processes
TLDR
By maintaining an adaptive data-dependent distribution over the underlying ODE, this model can successfully capture the dynamics of low-dimensional systems from just a few data-points and scale up to challenging high-dimensional time-series with unknown latent dynamics such as rotating MNIST digits.
How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization
TLDR
This paper introduces a theoretically-grounded combination of both optimal transport and stability regularizations which encourage neural ODEs to prefer simpler dynamics out of all the dynamics that solve a problem well, leading to faster convergence and to fewer discretizations of the solver.
Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
TLDR
This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-e finite-parameter training and tunable precision of Neural ODEs.
On Second Order Behaviour in Augmented Neural ODEs
TLDR
This work shows how the adjoint sensitivity method can be extended to SONODEs and proves that the optimisation of a first order coupled ODE is equivalent and computationally more efficient, and extends the theoretical understanding of the broader class of Augmented NODEs by showing they can also learn higher order dynamics with a minimal number of augmented dimensions, but at the cost of interpretability.
...
...

References

SHOWING 1-10 OF 34 REFERENCES
Neural Ordinary Differential Equations
TLDR
This work shows how to scalably backpropagate through any ODE solver, without access to its internal operations, which allows end-to-end training of ODEs within larger models.
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
TLDR
This paper uses Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density and demonstrates the approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
TLDR
Deep Variational Bayes Filters is introduced, a new method for unsupervised learning and identification of latent Markovian state space models that can overcome intractable inference distributions via variational inference and enables realistic long-term prediction.
Sequential Neural Models with Stochastic Layers
TLDR
Stochastic recurrent neural networks are introduced which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model.
Deep Temporal Sigmoid Belief Networks for Sequence Modeling
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks
Gaussian Process Prior Variational Autoencoders
TLDR
A new model is introduced, the Gaussian Process (GP) Prior Variational Autoencoder (GPPVAE), which aims to combine the power of VAEs with the ability to model correlations afforded by GP priors, and leverages structure in the covariance matrix to achieve efficient inference in this new class of models.
Auto-Encoding Variational Bayes
TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
Learning unknown ODE models with Gaussian processes
TLDR
This work proposes to learn non-linear, unknown differential functions from state observations using Gaussian process vector fields within the exact ODE formalism and demonstrates the model's capabilities to infer dynamics from sparse data and to simulate the system forward into future.
An Information-Theoretic Analysis of Deep Latent-Variable Models
TLDR
An information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference and how this framework sheds light on many recent proposed extensions to the variational autoencoder family is presented.
Disentangled Sequential Autoencoder
TLDR
Empirical evidence is given for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.
...
...