Transcoding compositionally: using attention to find more generalizable solutions

@article{Korrel2019TranscodingCU,
  title={Transcoding compositionally: using attention to find more generalizable solutions},
  author={K. Korrel and Dieuwke Hupkes and Verna Dankers and Elia Bruni},
  journal={ArXiv},
  year={2019},
  volume={abs/1906.01234}
}
  • K. Korrel, Dieuwke Hupkes, +1 author Elia Bruni
  • Published 2019
  • Computer Science
  • ArXiv
  • While sequence-to-sequence models have shown remarkable generalization power across several natural language tasks, their construct of solutions are argued to be less compositional than human-like generalization. In this paper, we present seq2attn, a new architecture that is specifically designed to exploit attention to find compositional patterns in the input. In seq2attn, the two standard components of an encoder-decoder model are connected via a transcoder, that modulates the information… CONTINUE READING
    12 Citations

    Figures, Tables, and Topics from this paper

    Explore Further: Topics Discussed in This Paper

    On the Realization of Compositionality in Neural Networks
    • 9
    • PDF
    Compositionality Decomposed: How do Neural Networks Generalise?
    • 28
    • PDF
    Assessing incrementality in sequence-to-sequence models
    • 2
    • PDF
    Compositionality as Directional Consistency in Sequential Neural Networks
    • PDF
    LSTMs Compose (and Learn) Bottom-Up
    • 1
    • PDF
    The compositionality of neural networks: integrating symbolism and connectionism
    • 15
    • PDF
    Word Interdependence Exposes How LSTMs Compose Representations
    • 3
    • PDF

    References

    SHOWING 1-10 OF 25 REFERENCES
    Learning compositionally through attentive guidance
    • 12
    • PDF
    On the Realization of Compositionality in Neural Networks
    • 9
    • PDF
    Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
    • 55
    • Highly Influential
    • PDF
    Memorize or generalize? Searching for a compositional RNN in a haystack
    • 31
    • Highly Influential
    • PDF
    Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks
    • 198
    • Highly Influential
    • PDF
    Attention is All you Need
    • 15,891
    • PDF
    Sequence to Sequence Learning with Neural Networks
    • 11,702
    • PDF
    Can LSTM Learn to Capture Agreement? The Case of Basque
    • 19
    • PDF
    Human-level concept learning through probabilistic program induction
    • 1,502
    • PDF