Corpus ID: 231802365

CKConv: Continuous Kernel Convolution For Sequential Data

@article{Romero2021CKConvCK,
  title={CKConv: Continuous Kernel Convolution For Sequential Data},
  author={David W. Romero and Anna Kuzina and Erik J. Bekkers and Jakub M. Tomczak and Mark Hoogendoorn},
  journal={ArXiv},
  year={2021},
  volume={abs/2102.02611}
}
Conventional neural architectures for sequential data present important limitations. Recurrent networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional networks are unable to handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that all these problems can be solved by formulating convolutional kernels in CNNs as continuous functions. The resulting Continuous… Expand
FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
TLDR
FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Expand
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
TLDR
A simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings, and incorporates and generalizes recent theory on continuous-time memorization to introduce a trainable subset of structured matrices A that endow LSSLs with long-range memory. Expand
Efficiently Modeling Long Sequences with Structured State Spaces
TLDR
This work proposes the Structured State Space (S4) sequence model based on a new parameterization for the SSM, and shows that it can be computed much more efficiently than prior approaches while preserving their theoretical strengths. Expand
Learning Equivariances and Partial Equivariances from Data
TLDR
Partial G-CNNs are introduced: a family of equivariant networks able to learn partial and full equivariances from data at every layer end-to-end and perform on par with G- CNNs when full equivariance is necessary, and outperform them otherwise. Expand
Neural Waveshaping Synthesis
We present the Neural Waveshaping Unit (NEWT): a novel, lightweight, fully causal approach to neural audio synthesis which operates directly in the waveform domain, with an accompanying optimisationExpand
Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups
TLDR
This work investigates the properties of representations learned by regular G-CNNs, and shows considerable parameter redundancy in group convolution kernels, which motivates further weight-tying by sharing convolution kernel over subgroups. Expand

References

SHOWING 1-10 OF 77 REFERENCES
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
TLDR
It is shown that an IndRNN can be easily regulated to prevent the gradient exploding and vanishing problems while allowing the network to learn long-term dependencies and work with non-saturated activation functions such as relu and be still trained robustly. Expand
Deep Parametric Continuous Convolutional Neural Networks
TLDR
The key idea is to exploit parameterized kernel functions that span the full continuous vector space, which allows us to learn over arbitrary data structures as long as their support relationship is computable. Expand
Dilated Recurrent Neural Networks
TLDR
This paper introduces a simple yet effective RNN connection structure, the DilatedRNN, characterized by multi-resolution dilated recurrent skip connections and introduces a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures. Expand
Learning Multiple Layers of Features from Tiny Images
TLDR
It is shown how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex, using a novel parallelization algorithm to distribute the work among multiple machines connected on a network. Expand
SchNet: A continuous-filter convolutional neural network for modeling quantum interactions
TLDR
This work proposes to use continuous-filter convolutional layers to be able to model local correlations without requiring the data to lie on a grid, and obtains a joint model for the total energy and interatomic forces that follows fundamental quantum-chemical principles. Expand
Mad Max: Affine Spline Insights Into Deep Learning
TLDR
A rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators is built and a simple penalty term is proposed that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other. Expand
Unitary Evolution Recurrent Neural Networks
TLDR
This work constructs an expressive unitary weight matrix by composing several structured matrices that act as building blocks with parameters to be learned, and demonstrates the potential of this architecture by achieving state of the art results in several hard tasks involving very long-term dependencies. Expand
Recurrent Neural Networks for Multivariate Time Series with Missing Values
TLDR
Novel deep learning models are developed based on Gated Recurrent Unit, a state-of-the-art recurrent neural network that takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Expand
Very deep convolutional neural networks for raw waveforms
TLDR
This work proposes very deep convolutional neural networks that directly use time-domain waveforms as inputs that are efficient to optimize over very long sequences, necessary for processing acoustic waveforms. Expand
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
TLDR
This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective, making truncated backpropagation feasible for long sequences and also improving full BPTT. Expand
...
1
2
3
4
5
...