• Corpus ID: 195766887

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

@article{LI2019EnhancingTL,
  title={Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting},
  author={SHIYANG LI and Xiaoyong Jin and Yao Xuan and Xiyou Zhou and Wenhu Chen and Yu-Xiang Wang and Xifeng Yan},
  journal={ArXiv},
  year={2019},
  volume={abs/1907.00235}
}
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. [...] Key Method In order to solve these two issues, we first propose convolutional self attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism.Expand
TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting
TLDR
The proposed tightly-coupled convolutional Transformer(TCCT) and three TCCT architectures could greatly improve the performance of existing state-of-art Transformer models on time series forecasting with much lower computation and memory costs, including canonical Transformer, LogTrans and Informer.
Transformers predicting the future. Applying attention in next-frame and time series forecasting
TLDR
This work is testing and evaluating the effectiveness of applying Transformer-like models on time series data, tackling susceptibility to anomalies, context awareness and space complexity by fine-tuning the hyperparameters, preprocessing the data, applying dimensionality reduction or convolutional encodings, etc.
AGCNT: Adaptive Graph Convolutional Network for Transformer-based Long Sequence Time-Series Forecasting
Long sequence time-series forecasting(LSTF) plays an important role in a variety of real-world application scenarios, such as electricity forecasting, weather forecasting, and traffic flow
Transformer with Sparse Attention Mechanism for Industrial Time Series Forecasting
Long sequence time-series forecasting has become a very important task in industrial data analysis. Recently, the Transformer model has been widely used in sequence processing tasks. However, because
NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting
TLDR
This work is the first attempt to propose a NonAutoregressive Transformer architecture for time series forecasting, aiming at overcoming the time delay and accumulative error issues in the canonical Transformer.
Adversarial Sparse Transformer for Time Series Forecasting
TLDR
Adversarial Sparse Transformer (AST) is proposed, a new time series forecasting model based on Generative Adversarial Networks (GANs), which adopts a SparseTransformer as the generator to learn a sparse attention map forTime series forecasting, and uses a discriminator to improve the prediction performance at a sequence level.
Parameter Efficient Deep Probabilistic Forecasting
TLDR
This work introduces a novel Bidirectional Temporal Convolutional Network (BiTCN), which requires an order of magnitude less parameters than a common Transformerbased approach and demonstrates that the model can be trained faster with significantly lower memory requirements, which reduces the infrastructure cost for deploying these models.
ChunkFormer: Learning Long Time Series with Multi-stage Chunked Transformer
TLDR
A novel architecture, ChunkFormer, is proposed that improves the existing Transformer framework to handle the challenges while dealing with long time series and gradually learns both local and global information without changing the total length of the input sequences.
Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting
TLDR
The Yformer model is proposed, based on a novel Y-shaped encoder-decoder architecture that uses direct connection from the downscaled encoder layer to the corresponding upsampled decoder layer in a U-Net inspired architecture, and stabilizes the encoding stacks with the addition of an auxiliary reconstruction loss.
Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
TLDR
This paper designs Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level and outperforms self-attention in both efficiency and accuracy.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 42 REFERENCES
Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks
TLDR
A novel deep learning framework, namely Long- and Short-term Time-series network (LSTNet), to address this open challenge of multivariate time series forecasting, using the Convolution Neural Network and the Recurrent Neural Network to extract short-term local dependency patterns among variables and to discover long-term patterns for time series trends.
Conditional Time Series Forecasting with Convolutional Neural Networks
TLDR
This paper compares the performance of the WaveNet model to a state-of-the-art fully convolutional network (FCN), and an autoregressive model popular in econometrics and shows that the model is much better able to learn important dependencies in between financial time series resulting in a more robust and accurate forecast.
Deep Learning: A Generic Approach for Extreme Condition Traffic Forecasting
TLDR
This work builds a deep neural network based on long short term memory (LSTM) units and applies Deep LSTM to forecast peak-hour traffic and manages to identify unique characteristics of the traffic data.
Multi-step Deep Autoregressive Forecasting with Latent States
In many forecasting applications, it is important to predict an overall trajectory within a period of time, i.e. a forecasting horizon. Various deep neural network models have been introduced to
Generating Long Sequences with Sparse Transformers
TLDR
This paper introduces sparse factorizations of the attention matrix which reduce this to $O(n)$, and generates unconditional samples that demonstrate global coherence and great diversity, and shows it is possible in principle to use self-attention to model sequences of length one million or more.
Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction
TLDR
This paper develops novel regularization schemes and uses scalable matrix factorization methods that are eminently suited for high-dimensional time series data that has many missing values, and makes interesting connections to graph regularization methods in the context of learning the dependencies in an autoregressive framework.
DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
TLDR
DeepAR is proposed, a methodology for producing accurate probabilistic forecasts, based on training an auto regressive recurrent network model on a large number of related time series, with accuracy improvements of around 15% compared to state-of-the-art methods.
Long-term Forecasting using Higher Order Tensor RNNs
TLDR
This work theoretically establishes the approximation guarantees and the variance bound for HOT-RNN for general sequence inputs, and demonstrates 5% ~ 12% improvements for long-term prediction over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world time series data.
Time-series Extreme Event Forecasting with Neural Networks at Uber
Accurate time-series forecasting during high variance segments (e.g., holidays), is critical for anomaly detection, optimal resource allocation, budget planning and other related tasks. At Uber
Long-term Forecasting using Tensor-Train RNNs
TLDR
Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics, and decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance.
...
1
2
3
4
5
...