• Computer Science
• Published in NAACL-HLT 2018
• DOI:10.18653/v1/N18-2074

Self-Attention with Relative Position Representations

@article{Shaw2018SelfAttentionWR,
  title={Self-Attention with Relative Position Representations},
  author={Peter Shaw and Jakob Uszkoreit and Ashish Vaswani},
  journal={ArXiv},
  year={2018},
  volume={abs/1803.02155}
}

• Peter Shaw, Jakob Uszkoreit, Ashish Vaswani

Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider… CONTINUE READING

View PDF on ArXiv

Save to Library

Create Alert

Cite