Corpus ID: 235358778

End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Tasks

  title={End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Tasks},
  author={Ananye Agarwal and P. Shenoy and Mausam},
Neural models and symbolic algorithms have recently been combined for tasks requiring both perception and reasoning. Neural models ground perceptual input into a conceptual vocabulary, on which a classical reasoning algorithm is applied to generate output. A key limitation is that such neural-to-symbolic models can only be trained end-to-end for tasks where the output space is symbolic. In this paper, we study neural-symbolic-neural models for reasoning tasks that require a conversion from an… Expand

Figures and Tables from this paper


Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
This work proposes a neural-symbolic visual question answering system that first recovers a structural scene representation from the image and a program trace from the question, then executes the program on the scene representation to obtain an answer. Expand
Neural-Symbolic Integration: A Compositional Perspective
It is shown that a symbolic module -- with any choice for syntax and semantics, as long as the deduction and abduction methods are exposed -- can be cleanly integrated with a neural module, and facilitate the latter's efficient training, achieving empirical performance that exceeds that of previous work. Expand
The Neuro-Symbolic Concept Learner: Interpreting Scenes Words and Sentences from Natural Supervision
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our modelExpand
Contrastive Learning of Structured World Models
These experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations. Expand
Deep Reasoning Networks for Unsupervised Pattern De-mixing with Constraint Reasoning
On CrystalStructure-Phase-Mapping, DRNets significantly outperform the state of the art and experts’ capabilities, recovering more precise and physically meaningful crystal structures. Expand
A Semantic Loss Function for Deep Learning with Symbolic Knowledge
A semantic loss function is derived from first principles that bridges between neural output vectors and logical constraints and significantly increases the ability of the neural network to predict structured objects, such as rankings and paths. Expand
Hybrid Classification and Reasoning for Image-based Constraint Solving
This paper explores the hybridization of classifying the images with the reasoning of a constraint solver, and shows that such hybrid approaches vastly outperform a separate approach, which encourages a further integration of prediction (probabilities) and constraint solving. Expand
SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver
This paper introduces a differentiable (smoothed) maximum satisfiability (MAXSAT) solver that can be integrated into the loop of larger deep learning systems and demonstrates that by integrating this solver into end-to-end learning systems, this approach shows promise in integrating logical structures within deep learning. Expand
A Primal Dual Formulation For Deep Learning With Constraints
This paper presents a constrained optimization formulation for training a deep network with a given set of hard constraints on output labels and shows that the constraints not only significantly reduce the number of constraint violations, but can also result in state-of-the-art performance. Expand
Inductive Logic Programming via Differentiable Deep Neural Logic Networks
It is shown that the proposed dNL-ILP solver supports desirable features such as recursion and predicate invention and outperforms the state of the art ILP solvers in classification tasks for Mutagenesis, Cora and IMDB datasets. Expand