A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

Abstract

We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multi-world approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.

Extracted Key Phrases

10 Figures and Tables

Showing 1-3 of 3 references

How many chairs are at the table?! H: wall M: 4! C: chair Q: What is the object on the chair?! H: pillow! M: floor

How many red chairs are there?! H: ()!

What is on the right side of cabinet?! H: picture M: bed! C: bed Q: What is on the wall?! H: mirror! M: bed! C: picture Q: What is beh H: lamp M: brown, pink C: picture

Showing 1-10 of 115 extracted citations
0501002014201520162017
Citations per Year

153 Citations

Semantic Scholar estimates that this publication has received between 122 and 198 citations based on the available data.

See our FAQ for additional information.