Corpus ID: 237940539

Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processing

@article{Han2021OscillatoryFN,
  title={Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processing},
  author={Bing Han and Cheng Wang and Kaushik Roy},
  journal={ArXiv},
  year={2021},
  volume={abs/2109.13090}
}
  • Bing Han, Cheng Wang, K. Roy
  • Published 14 September 2021
  • Computer Science
  • ArXiv
Tremendous progress has been made in sequential processing with the recent advances in recurrent neural networks. However, recurrent architectures face the challenge of exploding/vanishing gradients during training, and require significant computational resources to execute back-propagation through time. Moreover, large models are typically needed for executing complex sequential tasks. To address these challenges, we propose a novel neuron model that has cosine activation with a time varying… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 36 REFERENCES
Learning Long Term Dependencies via Fourier Recurrent Units
TLDR
A simple recurrent architecture, the Fourier Recurrent Unit (FRU), that stabilizes the gradients that arise in its training while giving us stronger expressive power, and demonstrates that with fewer parameters the proposed architecture outperforms other recurrent architectures on many tasks. Expand
Dilated Recurrent Neural Networks
TLDR
This paper introduces a simple yet effective RNN connection structure, the DilatedRNN, characterized by multi-resolution dilated recurrent skip connections and introduces a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures. Expand
Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
TLDR
This work proposes a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices by parametrizing with a skew-symmetric matrix using the Cayley transform. Expand
ForeNet : fourier recurrent networks for time series prediction
TLDR
A new interpretation to recurrent networks for time series prediction is linked together to form a new network called Fourier Recurrent Network (ForeNet), which speeds up the learning, and the generalization performance is superior to traditional recurrent network. Expand
AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
TLDR
This paper draws connections between recurrent networks and ordinary differential equations and proposes a special form of recurrent networks called AntisymmetricRNN, able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Expand
Unitary Evolution Recurrent Neural Networks
TLDR
This work constructs an expressive unitary weight matrix by composing several structured matrices that act as building blocks with parameters to be learned, and demonstrates the potential of this architecture by achieving state of the art results in several hard tasks involving very long-term dependencies. Expand
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
TLDR
This paper proposes a simpler solution that use recurrent neural networks composed of rectified linear units that is comparable to LSTM on four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem. Expand
Improvement of Learning in Recurrent Networks by Substituting the Sigmoid Activation Function
TLDR
This work has shown that full gradient learning schemes, such as RTRL and BPTT, have been proposed for learning in fully-connected recurrent networks and can also be used to train the weights of the recurrent hidden layer in SRNs. Expand
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
TLDR
The Skip RNN model is introduced which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph, which can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline Rnn models. Expand
...
1
2
3
4
...