Interactive Grounded Language Acquisition and Generalization in a 2D World

@article{Yu2018InteractiveGL,
  title={Interactive Grounded Language Acquisition and Generalization in a 2D World},
  author={Haonan Yu and Haichao Zhang and Wei Xu},
  journal={CoRR},
  year={2018},
  volume={abs/1802.01433}
}
We build a virtual agent for learning language in a 2D maze-like world. The agent sees images of the surrounding environment, listens to a virtual teacher, and takes actions to receive rewards. It interactively learns the teacher’s language from scratch based on two language use cases: sentence-directed navigation and question answering. It learns simultaneously the visual representations of the world, the language, and the action control. By disentangling language grounding from other… CONTINUE READING
36
Twitter Mentions

Citations

Publications citing this paper.
SHOWING 1-10 OF 16 CITATIONS

A Survey of Reinforcement Learning Informed by Natural Language

Jelena Luketina, Nantas Nardelli, +5 authors Tim Rocktäschel
  • ArXiv
  • 2019
VIEW 2 EXCERPTS
CITES METHODS

Attention Based Natural Language Grounding by Navigating Virtual Environment

  • 2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
  • 2018
VIEW 1 EXCERPT
CITES BACKGROUND

References

Publications referenced by this paper.
SHOWING 1-10 OF 52 REFERENCES

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

  • 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • 2018
VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

Bidirectional recurrent neural networks

  • IEEE Trans. Signal Processing
  • 1997
VIEW 11 EXCERPTS
HIGHLY INFLUENTIAL

Stacked Attention Networks for Image Question Answering

  • 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2016
VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

VQA: Visual Question Answering

VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

Driving Under the Influence (of Language)

  • IEEE Transactions on Neural Networks and Learning Systems
  • 2018
VIEW 1 EXCERPT

Similar Papers

Loading similar papers…