Liquid Structural State-Space Models
@article{Hasani2022LiquidSS, title={Liquid Structural State-Space Models}, author={Ramin M. Hasani and Mathias Lechner and Tsun-Hsuan Wang and Makram Chahine and Alexander Amini and Daniela Rus}, journal={ArXiv}, year={2022}, volume={abs/2209.12951} }
A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structural SSM such as S4 is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural…
6 Citations
Simplified State Space Layers for Sequence Modeling
- Computer ScienceArXiv
- 2022
A state space layer that can leverage e-cient and widely implemented parallel scans, allowing S5 to match the computational e-ciency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks.
Gated Recurrent Neural Networks with Weighted Time-Delay Feedback
- Computer ScienceArXiv
- 2022
Results show that τ -GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.
Closed-form continuous-time neural networks
- Computer ScienceNature Machine Intelligence
- 2022
It is shown that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form and obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts.
Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap
- Computer ScienceArXiv
- 2022
The results imply that the causality gap can be solved in situation one with the proposed training guideline with any modern network architecture, whereas achieving out-of-distribution generalization requires further investigations, for instance, on data diversity rather than the model architecture.
On the Forward Invariance of Neural ODEs
- Computer Science, MathematicsArXiv
- 2022
The invariance propagation is demonstrated on a comprehensive series of representation learning tasks, including spiral curve regression, autoregressive modeling of joint physical dynamics, convexity portrait of a function, and safe neural control of collision avoidance for autonomous vehicles.
Interpreting Neural Policies with Disentangled Tree Representations
- Computer ScienceArXiv
- 2022
A new algorithm is designed that programmatically extracts tree representations from compact neural policies, in the form of a set of logic programs grounded by world state, that allows for interpretability metrics that measure disentanglement of learned neural dynamics from a concentration of decisions, mutual information and modularity perspectives.
References
SHOWING 1-10 OF 96 REFERENCES
Diagonal State Spaces are as Effective as Structured State Spaces
- Computer ScienceArXiv
- 2022
The Diagonal State Space (DSS) model matches the performance of S4 on Long Range Arena tasks, speech classification on Speech Commands dataset, while being conceptually simpler and straightforward to implement.
On the Parameterization and Initialization of Diagonal State Space Models
- Computer ScienceArXiv
- 2022
A simple diagonal version of S4 whose kernel computation requires just 2 lines of code and performs comparably to S4 in almost all settings, with state-of-the-art results for image, audio, and medical time-series domains, and averaging 85% on the Long Range Arena benchmark.
Simplified State Space Layers for Sequence Modeling
- Computer ScienceArXiv
- 2022
A state space layer that can leverage e-cient and widely implemented parallel scans, allowing S5 to match the computational e-ciency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks.
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
- Computer ScienceNeurIPS
- 2021
A simple sequence model inspired by control systems that generalizes RNN heuristics, temporal convolutions, and neural differential equations while addressing their shortcomings, and incorporates and generalizes recent theory on continuous-time memorization to introduce a trainable subset of structured matrices A that endow LSSLs with long-range memory.
Efficiently Modeling Long Sequences with Structured State Spaces
- Computer ScienceICLR
- 2022
The Structured State Space sequence model (S4) is proposed, based on a new parameterization for the SSM, and it is shown that it can be computed much more efficiently than prior approaches while preserving their theoretical strengths.
Learning Long-Term Dependencies in Irregularly-Sampled Time Series
- Computer ScienceNeurIPS
- 2020
This work designs a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path.
How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections
- Computer ScienceArXiv
- 2022
A more general and intuitive formulation of the HiPPO framework is derived, which provides a simple mathematical interpretation of S4 as a decomposition onto exponentially-warped Legendre polynomials, explaining its ability to capture long dependencies.
State-Regularized Recurrent Neural Networks
- Computer ScienceICML
- 2019
It is shown that state-regularization simplifies the extraction of finite state automata modeling an RNN's state transition dynamics and forces RNNs to operate more like automata with external memory and less like finite state machines, which makes Rnns have better interpretability and explainability.
RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?
- Computer ScienceICLR
- 2020
GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
- Computer ScienceNeurIPS
- 2019
Empirical evaluation shows that the proposed GRU-ODE-Bayes method outperforms the state of the art on both synthetic data and real-world data with applications in healthcare and climate forecast and the continuity prior is shown to be well suited for low number of samples settings.