Liquid Structural State-Space Models

@article{Hasani2022LiquidSS,
  title={Liquid Structural State-Space Models},
  author={Ramin M. Hasani and Mathias Lechner and Tsun-Hsuan Wang and Makram Chahine and Alexander Amini and Daniela Rus},
  journal={ArXiv},
  year={2022},
  volume={abs/2209.12951}
}
A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structural SSM such as S4 is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural… 

Figures and Tables from this paper

Simplified State Space Layers for Sequence Modeling

A state space layer that can leverage e-cient and widely implemented parallel scans, allowing S5 to match the computational e-ciency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks.

Gated Recurrent Neural Networks with Weighted Time-Delay Feedback

Results show that τ -GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.

Closed-form continuous-time neural networks

It is shown that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form and obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts.

Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap

The results imply that the causality gap can be solved in situation one with the proposed training guideline with any modern network architecture, whereas achieving out-of-distribution generalization requires further investigations, for instance, on data diversity rather than the model architecture.

On the Forward Invariance of Neural ODEs

The invariance propagation is demonstrated on a comprehensive series of representation learning tasks, including spiral curve regression, autoregressive modeling of joint physical dynamics, convexity portrait of a function, and safe neural control of collision avoidance for autonomous vehicles.

Interpreting Neural Policies with Disentangled Tree Representations

A new algorithm is designed that programmatically extracts tree representations from compact neural policies, in the form of a set of logic programs grounded by world state, that allows for interpretability metrics that measure disentanglement of learned neural dynamics from a concentration of decisions, mutual information and modularity perspectives.

References

SHOWING 1-10 OF 96 REFERENCES

Diagonal State Spaces are as Effective as Structured State Spaces

The Diagonal State Space (DSS) model matches the performance of S4 on Long Range Arena tasks, speech classification on Speech Commands dataset, while being conceptually simpler and straightforward to implement.

On the Parameterization and Initialization of Diagonal State Space Models

A simple diagonal version of S4 whose kernel computation requires just 2 lines of code and performs comparably to S4 in almost all settings, with state-of-the-art results for image, audio, and medical time-series domains, and averaging 85% on the Long Range Arena benchmark.

Simplified State Space Layers for Sequence Modeling

A state space layer that can leverage e-cient and widely implemented parallel scans, allowing S5 to match the computational e-ciency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks.

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

A simple sequence model inspired by control systems that generalizes RNN heuristics, temporal convolutions, and neural differential equations while addressing their shortcomings, and incorporates and generalizes recent theory on continuous-time memorization to introduce a trainable subset of structured matrices A that endow LSSLs with long-range memory.

Efficiently Modeling Long Sequences with Structured State Spaces

The Structured State Space sequence model (S4) is proposed, based on a new parameterization for the SSM, and it is shown that it can be computed much more efficiently than prior approaches while preserving their theoretical strengths.

Learning Long-Term Dependencies in Irregularly-Sampled Time Series

This work designs a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path.

How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections

A more general and intuitive formulation of the HiPPO framework is derived, which provides a simple mathematical interpretation of S4 as a decomposition onto exponentially-warped Legendre polynomials, explaining its ability to capture long dependencies.

State-Regularized Recurrent Neural Networks

It is shown that state-regularization simplifies the extraction of finite state automata modeling an RNN's state transition dynamics and forces RNNs to operate more like automata with external memory and less like finite state machines, which makes Rnns have better interpretability and explainability.

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

Empirical evaluation shows that the proposed GRU-ODE-Bayes method outperforms the state of the art on both synthetic data and real-world data with applications in healthcare and climate forecast and the continuity prior is shown to be well suited for low number of samples settings.
...