• DOI:10.18653/v1/2021.emnlp-main.141
• Corpus ID: 243865433

Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text

@article{Clark2021IconaryAP,
  title={Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text},
  author={Christopher Clark and Jordi Salvador and Dustin Schwenk and Derrick Bonafilia and Mark Yatskar and Eric Kolve and Alvaro Herrasti and Jonghyun Choi and Sachin Mehta and Sam Skjonsberg and Carissa Schoenick and Aaron Sarnat and Hannaneh Hajishirzi and Aniruddha Kembhavi and Oren Etzioni and Ali Farhadi},
  journal={ArXiv},
  year={2021},
  volume={abs/2112.00800}
}

• Christopher Clark, Jordi Salvador, +13 authors Ali Farhadi
• Published 1 December 2021
• Computer Science
• ArXiv

Communicating with humans is challenging for AIs because it requires a shared understanding of the world, complex semantics (e.g., metaphors or analogies), and at times multi-modal gestures (e.g., pointing with a finger, or an arrow in a diagram). We investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community. In Iconary, a Guesser tries to identify a phrase that a Drawer is drawing by composing icons, and the Drawer iteratively revises the drawing to help the Guesser in response. This back-and-forth often uses canonical scenes, visual metaphor, or icon compositions to express challenging words, making it an ideal test for mixing language and visual/symbolic communication in AI. We propose models to play Iconary and train them on over 55,000 games between human players. Our models are skillful players and are able to employ world knowledge in language models to play with words unseen during training.