Going Beyond Approximation: Encoding Constraints for Explainable Multi-hop Inference via Differentiable Combinatorial Solvers

  title={Going Beyond Approximation: Encoding Constraints for Explainable Multi-hop Inference via Differentiable Combinatorial Solvers},
  author={Mokanarangan Thayaparan and Marco Valentino and Andr{\'e} Freitas},
Integer Linear Programming (ILP) provides a viable mechanism to encode explicit and controllable assumptions about explainable multi-hop inference with natural language. How-ever, an ILP formulation is non-differentiable and cannot be integrated into broader deep learning architectures. Recently, Thayaparan et al. (2021a) proposed a novel methodology to integrate ILP with Transformers to achieve end-to-end differentiability for complex multi-hop inference. While this hybrid framework has been… 

Figures and Tables from this paper



Diff-Explainer: Differentiable Convex Optimization for Explainable Multi-hop Inference

The first hybrid framework for explainable multi-hop inference that integrates explicit constraints with neural architectures through differentiable convex optimization is presented, which allows for the fine-tuning of neural representations within a constrained optimization framework to answer and explainMulti-hop questions in natural language.

Hybrid Autoregressive Inference for Scalable Multi-Hop Explanation Regeneration

This paper presents SCAR (for Scalable Autoregressive Inference), a hybrid framework that iteratively combines a Transformer-based bi-encoder with a sparse model of explanatory power, designed to leverage explicit inference patterns in the explanations.

CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraints

This work aims to integrate integer programming solvers into neural network architectures as layers capable of learning both the cost terms and the constraints, and demonstrates the potential of such layers with an extensive performance analysis on synthetic data and with a demonstration on a competitive computer vision keypoint matching benchmark.

Differentiable Convex Optimization Layers

This paper introduces disciplined parametrized programming, a subset of disciplined convex programming, and demonstrates how to efficiently differentiate through each of these components, allowing for end-to-end analytical differentiation through the entire convex program.

Differentiation of Blackbox Combinatorial Solvers

This work presents a method that implements an efficient backward pass through blackbox implementations of combinatorial solvers with linear objective functions, and incorporates the Gurobi MIP solver, Blossom V algorithm, and Dijkstra's algorithm into architectures that extract suitable features from raw inputs for the traveling salesman problem, the min-cost perfect matching problem and the shortest path problem.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.

Explainable Inference Over Grounding-Abstract Chains for Science Questions

This paper frames question answering as a natural language abductive reasoning problem, constructing plausible explanations for each candidate answer and then selecting the candidate with the best explanation as the final answer by employing a linear programming formalism.

PRover: Proof Generation for Interpretable Reasoning over Rules

This work proposes PROVER, an interpretable transformer-based model that jointly answers binary questions over rule-bases and generates the corresponding proofs, and learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm.

Question Answering via Integer Programming over Semi-Structured Knowledge

This work proposes a structured inference system for this task, formulated as an Integer Linear Program (ILP), that answers natural language questions using a semi-structured knowledge base derived from text, including questions requiring multi-step inference and a combination of multiple facts.

MPNet: Masked and Permuted Pre-training for Language Understanding

This paper proposes MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods.