A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

Abstract

We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multi-world approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.

Extracted Key Phrases

Showing 1-3 of 3 references

How many chairs are at the table?! H: wall M: 4! C: chair Q: What is the object on the chair?! H: pillow! M: floor

How many red chairs are there?! H: ()!

What is on the right side of cabinet?! H: picture M: bed! C: bed Q: What is on the wall?! H: mirror! M: bed! C: picture Q: What is beh H: lamp M: brown, pink C: picture

Showing 1-10 of 99 extracted citations
0502014201520162017
Citations per Year

136 Citations

Semantic Scholar estimates that this publication has received between 106 and 182 citations based on the available data.

See our FAQ for additional information.