Semantic Image Manipulation Using Scene Graphs

@article{Dhamo2020SemanticIM,
  title={Semantic Image Manipulation Using Scene Graphs},
  author={Helisa Dhamo and Azade Farshad and Iro Laina and N. Navab and Gregory Hager and Federico Tombari and C. Rupprecht},
  journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
  pages={5212-5221}
}
Image manipulation can be considered a special case of image generation where the image to be produced is a modification of an existing image. Image generation and manipulation have been, for the most part, tasks that operate on raw pixels. However, the remarkable progress in learning rich image and object representations has opened the way for tasks such as text-to-image or layout-to-image generation that are mainly driven by semantics. In our work, we address the novel problem of image… Expand
Scene Graphs: A Survey of Generations and Applications
TLDR
This survey conducts a comprehensive investigation of the current scene graph research and conducted a comprehensive and systematic discussion on the generation method of the scene graph (SGG) and the SGG with the aid of prior knowledge. Expand
Unconditional Scene Graph Generation
TLDR
This work develops a deep auto-regressive model called SceneGraphGen which can directly learn the probability distribution over labelled and directed graphs using a hierarchical recurrent architecture and demonstrates the application of the generated graphs in image synthesis, anomaly detection and scene graph completion. Expand
Image-Graph-Image Translation via Auto-Encoding
TLDR
This work presents the first convolutional neural network that learns an image-to-graph translation task without needing external supervision, and is the first to present a self-supervised approach based on a fully-differentiable auto-encoder in which the bottleneck encodes the graph's nodes and edges. Expand
Scene Graph Embeddings Using Relative Similarity Supervision
TLDR
This work proposes a novel loss function that operates on pairs of similar and dissimilar images and imposes relative ordering between them in embedding space and demonstrates that this Ranking loss leads to robust representations that outperform well-known contrastive losses on the retrieval task. Expand
Learning to Generate Scene Graph from Natural Language Supervision
  • Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, Yin Li
  • Computer Science
  • 2021
Learning from image-text data has demonstrated recent success for many recognition tasks, yet is currently limited to visual features or individual visual concepts such as objects. In this paper, weExpand
A Novel Approach to Scene Graph Vectorization
In recent times due to the advancement in perceptual applications, focus in computer vision has been inclined towards tasks that require a significant level of semantic understanding of scenes.Expand
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions
TLDR
This work proposes a learned method that regresses a scene graph from the point cloud of a scene, based on PointNet and Graph Convolutional Networks, and introduces 3DSSG, a semiautomatically generated dataset, that contains semantically rich scene graphs of 3D scenes. Expand
FashionGraph: Understanding fashion data using scene graph generation
TLDR
This paper proposes to generate scene graphs for existing fashion datasets by detecting relationships between fashion objects, their parts, and their attributes and shows various benefits for fashion applications by exploiting scene graph knowledge. Expand
Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs
Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Thereby, these specifications should be abstract, i.e. allowing easy user interaction,Expand
Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation
The recent progress in learning image feature representations has opened the way for tasks such as label-to-image or text-to-image synthesis. However, one particular challenge widely observed inExpand
...
1
2
...

References

SHOWING 1-10 OF 65 REFERENCES
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
TLDR
This work proposes a novel hierarchical approach for text-to-image synthesis by inferring semantic layout and shows that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches. Expand
Specifying Object Attributes and Relations in Interactive Scene Generation
  • Oron Ashual, L. Wolf
  • Computer Science
  • 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
  • 2019
TLDR
The method separates between a layout embedding and an appearance embedding, which leads to generated images that better match the scene graph, have higher visual quality, and support more complex scene graphs. Expand
Image retrieval using scene graphs
TLDR
A conditional random field model that reasons about possible groundings of scene graphs to test images and shows that the full model can be used to improve object localization compared to baseline methods and outperforms retrieval methods that use only objects or low-level image features. Expand
PatchNet: a patch-based image representation for interactive library-driven image editing
TLDR
It is shown how this PatchNet representation can be used as a basis for interactive, library-driven, image editing, allowing suitable regions to be found and applied in a few seconds, even from a library containing thousands of images. Expand
Image Generation from Scene Graphs
TLDR
This work proposes a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships, and validates this approach on Visual Genome and COCO-Stuff. Expand
Image Generation From Layout
TLDR
The proposed Layout2Im model significantly outperforms the previous state of the art, boosting the best reported inception score by 24.66% and 28.57% on the very challenging COCO-Stuff and Visual Genome datasets, respectively. Expand
Scene Graph Generation by Iterative Message Passing
TLDR
This work explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image, and proposes a novel end-to-end model that generates such structured scene representation from an input image. Expand
Attentive Relational Networks for Mapping Images to Scene Graphs
TLDR
A novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem, and accurate scene graphs are produced by the relation inference module to recognize all entities and corresponding relations. Expand
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
TLDR
A subgraph-based connection graph is proposed to concisely represent the scene graph during the inference to improve the efficiency of scene graph generation and outperforms the state-of-the-art method in both accuracy and speed. Expand
Pixels to Graphs by Associative Embedding
TLDR
A method for training a convolutional neural network such that it takes in an input image and produces a full graph definition and is done end-to-end in a single stage with the use of associative embeddings. Expand
...
1
2
3
4
5
...