Self-Attention with Relative Position Representations

@inproceedings{Shaw2018SelfAttentionWR,
  title={Self-Attention with Relative Position Representations},
  author={Peter Shaw and Jakob Uszkoreit and Ashish Vaswani},
  booktitle={NAACL-HLT},
  year={2018}
}
Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider… Expand
View on ACL
arxiv.org
543 Citations
Highly Influential Citations
99
Background Citations
257
Methods Citations
238
Results Citations
15

Supplemental Code

Github Repo
Via Papers with Code
OpenNMT/OpenNMT-tf
Neural machine translation and sequence learning using TensorFlow
1171 Stars • 337 Forks
Github Repo
Via Papers with Code
tensorflow/tensor2tensor
Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
10552 Stars • 2710 Forks

543 Citations

Citation Type

Citation Type

Self-Attention with Structural Position Representations
Improve Transformer Models with Better Relative Position Embeddings
Context-Aware Self-Attention Networks
Position-aware self-attention based neural sequence labeling
An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation
Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention
Self-Attention and Dynamic Convolution Hybrid Model for Neural Machine Translation
Assessing the Ability of Self-Attention Networks to Learn Word Order
Joint Source-Target Self Attention with Locality Constraints
Pointwise Conv Projection Shared Projection Output Representations Attention Input Representations
  • 2019
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 17 REFERENCES
Effective Approaches to Attention-based Neural Machine Translation
Attention is All you Need
End-To-End Memory Networks
Graph Attention Networks
Rethinking the Inception Architecture for Computer Vision
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Sequence to Sequence Learning with Neural Networks
Neural Machine Translation in Linear Time
Neural Machine Translation by Jointly Learning to Align and Translate
Convolutional Sequence to Sequence Learning
...
1
2
...