Skip to search formSkip to main contentSkip to account menu

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

This work proposes a technique for producing ‘visual explanations’ for decisions from a large class of Convolutional Neural Network (CNN)-based models, making them more transparent and explainable, and shows that even non-attention based models learn to localize discriminative regions of input image.
View on IEEE (opens in a new tab)

Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization

It is shown that Guided Grad-CAM helps untrained users successfully discern a "stronger" deep network from a "weaker" one even when both networks make identical predictions, and also exposes the somewhat surprising insight that common CNN + LSTM models can be good at localizing discriminative input image regions despite not being trained on grounded image-text pairs.

Embodied Question Answering

A new AI task where an agent is spawned at a random location in a 3D environment and asked a question ('What color is the car?'), and the agent must first intelligently navigate to explore the environment, gather necessary visual information through first-person (egocentric) vision, and then answer the question.
View on IEEE (opens in a new tab)

Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning

This work poses a cooperative ‘image guessing’ game between two agents who communicate in natural language dialog so that Q-BOT can select an unseen image from a lineup of images and shows the emergence of grounded language and communication among ‘visual’ dialog agents with no human supervision.
View on IEEE (opens in a new tab)

TarMAC: Targeted Multi-Agent Communication

This work proposes a targeted communication architecture for multi-agent reinforcement learning, where agents learn both what messages to send and whom to address them to while performing cooperative tasks in partially-observable environments, and augment this with a multi-round communication approach.
View PDF on arXiv (opens in a new tab)

Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?

The VQA-HAT (Human ATtention) dataset is introduced and attention maps generated by state-of-the-art V QA models are evaluated against human attention both qualitatively and quantitatively.
View on ACL (opens in a new tab)

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

This work adapts the recently proposed ViLBERT model for multi-turn visually-grounded conversations and finds that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG but hurts MRR, highlighting a trade-off between the two primary metrics.
View PDF on arXiv (opens in a new tab)

End-to-end Audio Visual Scene-aware Dialog Using Multimodal Attention-based Video Features

This paper introduces a new data set of dialogs about videos of human behaviors, as well as an end-to-end Audio Visual Scene-Aware Dialog (AVSD) model, trained using thisnew data set, that generates responses in a dialog about a video.
View on IEEE (opens in a new tab)

Neural Modular Control for Embodied Question Answering

This work uses imitation learning to warm-start policies at each level of the hierarchy, dramatically increasing sample efficiency, followed by reinforcement learning, for learning policies for navigation over long planning horizons from language input.
View PDF on arXiv (opens in a new tab)

Embodied Question Answering in Photorealistic Environments With Point Cloud Perception

It is found that point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating the use (and continued study) of 3D deep learning models for embodied navigation.
View on IEEE (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)