# Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections

@article{Tth2021Seq2TensAE, title={Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections}, author={Csaba T{\'o}th and Patric Bonnier and Harald Oberhauser}, journal={ArXiv}, year={2021}, volume={abs/2006.07027} }

Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies. At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning. We use a classical mathematical object -- the tensor algebra -- to capture such dependencies. To address the innate computational complexity of high degree tensors, we use compositions of low-rank tensor projections. This…

## Figures and Tables from this paper

## 6 Citations

### Capturing Graphs with Hypo-Elliptic Diffusions

- Computer ScienceArXiv
- 2022

This work develops a novel tensor-valued graph operator, which it calls the hypo-elliptic graph Laplacian, which provides theoretical guarantees and efficient low-rank approximation algorithms.

### Proper Scoring Rules, Gradients, Divergences, and Entropies for Paths and Time Series

- Computer Science
- 2021

This work uses the statistical framework of proper scoring rules with classical mathematical results to derive a principled approach to decision making with forecasts of gradients, entropy, and divergence that are tailor-made to respect the underlying non-Euclidean structure.

### Generalized iterated-sums signatures

- Mathematics, Computer ScienceArXiv
- 2020

The algebraic properties of a generalized version of the iterated-sums signature are explored, inspired by previous work of F.~Kiraly and H.~Oberhauser, and the character property of the associated linear map over the tensor algebra is recovered.

### Adjusting for Autocorrelated Errors in Neural Networks for Time Series Regression and Forecasting

- Computer ScienceNeurIPS
- 2021

This paper proposes to learn the autocorrelation coefficient jointly with the model parameters in order to adjust for Autocorrelated errors, and proposes to broaden the method to time series forecasting and applies it with various state-of-the-art models.

### Tropical time series, iterated-sums signatures and quasisymmetric functions

- Mathematics, Computer ScienceArXiv
- 2020

This work introduces the iterated-sums signature over any commutative semiring, as it leads to features of real-valued time series that are not easily available using existing signature-type objects.

### CHRONOSIG: Digital Triage for Secondary Mental Healthcare using Natural Language Processing - Rationale and Protocol

- MedicinemedRxiv
- 2021

Background: Accessing specialist secondary mental health care in the NHS in England requires a referral, usually from primary or acute care. Community mental health teams triage these referrals…

## References

SHOWING 1-10 OF 106 REFERENCES

### Learning Efficient Tensor Representations with Ring-structured Networks

- Computer ScienceICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- 2019

It is shown that the structure information and high-order correlations within a 2D image can be captured efficiently by employing an appropriate tensorization and TR decomposition.

### On the Expressive Power of Deep Learning: A Tensor Analysis

- Computer ScienceCOLT 2016
- 2015

It is proved that besides a negligible set, all functions that can be implemented by a deep network of polynomial size, require exponential size in order to be realized (or even approximated) by a shallow network.

### Tensorizing Neural Networks

- Computer ScienceNIPS
- 2015

This paper converts the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved.

### Tensor-Train Decomposition

- Computer ScienceSIAM J. Sci. Comput.
- 2011

The new form gives a clear and convenient way to implement all basic operations efficiently, and the efficiency is demonstrated by the computation of the smallest eigenvalue of a 19-dimensional operator.

### Disentangled Sequential Autoencoder

- Computer ScienceICML
- 2018

Empirical evidence is given for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.

### Generalized Tensor Models for Recurrent Neural Networks

- Computer ScienceICLR
- 2019

This work attempts to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and shows that they also benefit from properties of universality and depth efficiency.

### Supervised Learning with Tensor Networks

- Computer ScienceNIPS
- 2016

It is demonstrated how algorithms for optimizing tensor networks can be adapted to supervised learning tasks by using matrix product states (tensor trains) to parameterize non-linear kernel learning models.

### Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions

- Computer ScienceFound. Trends Mach. Learn.
- 2016

A focus is on the Tucker and tensor train TT decompositions and their extensions, and on demonstrating the ability of tensor network to provide linearly or even super-linearly e.g., logarithmically scalablesolutions, as illustrated in detail in Part 2 of this monograph.

### Bayesian Learning from Sequential Data using Gaussian Processes with Signature Covariances

- Computer ScienceICML
- 2020

We develop a Bayesian approach to learning from sequential data by using Gaussian processes (GPs) with so-called signature kernels as covariance functions. This allows to make sequences of different…

### Sequence to Sequence Learning with Neural Networks

- Computer ScienceNIPS
- 2014

This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.