Neural Motifs: Scene Graph Parsing with Global Context

@article{Zellers2017NeuralMS,
  title={Neural Motifs: Scene Graph Parsing with Global Context},
  author={Rowan Zellers and Mark Yatskar and Sam Thomson and Yejin Choi},
  journal={2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2017},
  pages={5831-5840}
}
We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at… CONTINUE READING

Figures, Tables, Results, and Topics from this paper.

Key Quantitative Results

  • This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain.
  • Finally, experiments show Stacked Motif Networks is effective at modeling global context, with a mean improvement of 2.9 points (7.1% relative improvement) over our new strong baseline.

Citations

Publications citing this paper.
SHOWING 1-10 OF 60 CITATIONS

A Study on Object Detection Technology for an Improved Visual Relationship Detection

Hyunji Choi, Yu-Jung Heo, Byung-Tak Zhang
  • 2019
VIEW 5 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

Exploring the Semantics for Visual Relationship Detection

VIEW 12 EXCERPTS
CITES METHODS, BACKGROUND & RESULTS
HIGHLY INFLUENCED

PANet: A Context Based Predicate Association Network for Scene Graph Generation

Yunian Chen, Yanjie Wang, Yang Zhang, Yanwen Guo
  • 2019 IEEE International Conference on Multimedia and Expo (ICME)
  • 2019
VIEW 10 EXCERPTS
CITES BACKGROUND, METHODS & RESULTS
HIGHLY INFLUENCED

Rethinking Visual Relationships for High-level Image Understanding

  • ArXiv
  • 2019
VIEW 10 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Scene Graph Prediction with Limited Labels

VIEW 7 EXCERPTS
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2018
2019

CITATION STATISTICS

  • 25 Highly Influenced Citations

  • Averaged 20 Citations per year from 2017 through 2019

  • 100% Increase in citations per year in 2019 over 2018

References

Publications referenced by this paper.
SHOWING 1-10 OF 59 REFERENCES

Scene Graph Generation by Iterative Message Passing

  • 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2017
VIEW 11 EXCERPTS
HIGHLY INFLUENTIAL

Long Short-Term Memory

  • Neural Computation
  • 1997
VIEW 16 EXCERPTS
HIGHLY INFLUENTIAL

Edge and Curve Detection for Visual Scene Analysis

  • IEEE Transactions on Computers
  • 1971
VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

Scene Graph Generation from Objects, Phrases and Region Captions

  • 2017 IEEE International Conference on Computer Vision (ICCV)
  • 2017
VIEW 12 EXCERPTS
HIGHLY INFLUENTIAL

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • 2015
VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

Highway long short-term memory RNNS for distant speech recognition

  • 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2015
VIEW 5 EXCERPTS
HIGHLY INFLUENTIAL