Talk the Walk: Navigating New York City through Grounded Dialogue
@article{Vries2018TalkTW, title={Talk the Walk: Navigating New York City through Grounded Dialogue}, author={H. D. Vries and Kurt Shuster and Dhruv Batra and D. Parikh and J. Weston and Douwe Kiela}, journal={ArXiv}, year={2018}, volume={abs/1807.03367} }
We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the… CONTINUE READING
Supplemental Code
Github Repo
Via Papers with Code
This repository provides code for reproducing experiments of the paper Talk The Walk: Navigating New York City Through Grounded Dialogue by Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela.
Figures, Tables, and Topics from this paper
Paper Mentions
65 Citations
AirDialogue: An Environment for Goal-Oriented Dialogue Research
- Computer Science
- EMNLP
- 2018
- 25
- Highly Influenced
- PDF
RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation
- Computer Science
- EMNLP/IJCNLP
- 2019
- 1
- PDF
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments
- Computer Science
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2019
- 100
- PDF
References
SHOWING 1-10 OF 54 REFERENCES
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
- Computer Science
- AAAI
- 2016
- 146
- PDF
Visual Dialog
- Computer Science
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- 2017
- 328
- PDF
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue
- Computer Science
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- 2017
- 229
- PDF
End-to-end optimization of goal-driven and visually grounded dialogue systems
- Computer Science
- IJCAI
- 2017
- 96
- PDF
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments
- Computer Science
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- 2018
- 312
- Highly Influential
- PDF
Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions
- Computer Science
- AAAI
- 2006
- 315
- PDF
Embodied Question Answering
- Computer Science
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- 2018
- 22
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
- Computer Science
- 2017 IEEE International Conference on Computer Vision (ICCV)
- 2017
- 283
- PDF
Emergent Language in a Multi-Modal, Multi-Step Referential Game
- Computer Science, Mathematics
- ArXiv
- 2017
- 29
- PDF