Corpus ID: 209323787

Axial Attention in Multidimensional Transformers

@article{Ho2019AxialAI,
  title={Axial Attention in Multidimensional Transformers},
  author={Jonathan Ho and Nal Kalchbrenner and Dirk Weissenborn and Tim Salimans},
  journal={ArXiv},
  year={2019},
  volume={abs/1912.12180}
}
  • Jonathan Ho, Nal Kalchbrenner, +1 author Tim Salimans
  • Published 2019
  • Computer Science
  • ArXiv
  • We propose Axial Transformers, a self-attention-based autoregressive model for images and other data organized as high dimensional tensors. Existing autoregressive models either suffer from excessively large computational resource requirements for high dimensional data, or make compromises in terms of distribution expressiveness or ease of implementation in order to decrease resource requirements. Our architecture, by contrast, maintains both full expressiveness over joint distributions over… CONTINUE READING

    Citations

    Publications citing this paper.
    SHOWING 1-5 OF 5 CITATIONS

    Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

    VIEW 3 EXCERPTS
    CITES BACKGROUND & METHODS

    Jukebox: A Generative Model for Music

    VIEW 1 EXCERPT

    MetNet: A Neural Weather Model for Precipitation Forecasting

    VIEW 2 EXCERPTS
    CITES METHODS & BACKGROUND

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 17 REFERENCES

    Image Transformer

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    Generating Long Sequences with Sparse Transformers

    VIEW 11 EXCERPTS
    HIGHLY INFLUENTIAL

    Generative Image Modeling Using Spatial LSTMs

    VIEW 1 EXCERPT

    Attention is All you Need

    VIEW 7 EXCERPTS
    HIGHLY INFLUENTIAL