Hypothesis Only Baselines in Natural Language Inference

@article{Poliak2018HypothesisOB,
  title={Hypothesis Only Baselines in Natural Language Inference},
  author={Adam Poliak and Jason Naradowsky and Aparajita Haldar and Rachel Rudinger and Benjamin Van Durme},
  journal={ArXiv},
  year={2018},
  volume={abs/1805.01042}
}
We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on 10 distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a… 

Figures and Tables from this paper

Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference

TLDR
This work analyzes hypothesis-only models trained on one of the recast datasets provided in Poliak et al.

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

TLDR
Two probabilistic methods are proposed to build models that are more robust to hypothesis-only biases in Natural Language Inference datasets and transfer better than a baseline architecture in 9 out of 12 NLI datasets.

Non-entailed subsequences as a challenge for natural language inference

Neural network models have shown great success at natural language inference (NLI), the task of determining whether a premise entails a hypothesis. However, recent studies suggest that these models

Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inference

Natural language inference (NLI) is among the most challenging tasks in natural language understanding. Recent work on unsupervised pretraining that leverages unsupervised signals such as

Annotation Artifacts in Natural Language Inference Data

TLDR
It is shown that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI, and that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes.

Improving Generalization by Incorporating Coverage in Natural Language Inference

TLDR
This work proposes to extend the input representations with an abstract view of the relation between the hypothesis and the premise, i.e., how well the individual words, or word n-grams, of the hypothesis are covered by the premise to improve generalization.

Relation Leakage in Elicited Natural Language Inference Datasets

TLDR
This work analyzes the problem of elicited sentence relation leakage in 8 modern NLI datasets, using a combination of previously established and novel model-based techniques, so as to enable ameliorating this leakage in futureNLI datasets.

HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference

TLDR
This work extracts various phrases from the hypotheses (artificial patterns) in the training sets, and shows that they have been strong indicators to the specific labels and investigates two debiasing approaches which exploit the artificial pattern modeling to mitigate such hypothesis-only bias.

Uncertain Natural Language Inference

TLDR
The feasibility of collecting annotations for UNLI is demonstrated by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise.

Open IE-Triples Inference– Corpora Development and DNN Architectures

  • Martin VitaP. Škoda
  • Computer Science
    2020 15th Conference on Computer Science and Information Systems (FedCSIS)
  • 2020
TLDR
This work proposes a method of transforming a general NLI corpus to an annotated corpus for relation inference that utilizes existing NLI annotations and introduces a novel relation inference corpus obtained from a well known SNLI corpus.
...

References

SHOWING 1-10 OF 58 REFERENCES

Annotation Artifacts in Natural Language Inference Data

TLDR
It is shown that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI, and that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes.

A large annotated corpus for learning natural language inference

TLDR
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.

Natural language inference

TLDR
This dissertation explores a range of approaches to NLI, beginning with methods which are robust but approximate, and proceeding to progressively more precise approaches, and greatly extends past work in natural logic to incorporate both semantic exclusion and implicativity.

Ordinal Common-sense Inference

TLDR
This work describes a framework for extracting common-sense knowledge from corpora, which is then used to construct a dataset for this ordinal entailment task, and annotates subsets of previously established datasets via the ordinal annotation protocol in order to analyze the distinctions between these and what is constructed.

A Phrase-Based Alignment Model for Natural Language Inference

TLDR
The MANLI system is presented, a new NLI aligner designed to address the alignment problem, which uses a phrase-based alignment representation, exploits external lexical resources, and capitalizes on a new set of supervised training data.

On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference

TLDR
A process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena is proposed, which finds its encoder appears most suited to supporting inferences at the syntax-semantics interface.

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

TLDR
The Multi-Genre Natural Language Inference corpus is introduced, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding and shows that it represents a substantially more difficult task than does the Stanford NLI corpus.

Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework

TLDR
A general strategy to automatically generate one or more sentential hypotheses based on an input sentence and pre-existing manual semantic annotations is presented, which enables us to probe a statistical RTE model’s performance on different aspects of semantics.

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

TLDR
It is shown how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks.

SciTaiL: A Textual Entailment Dataset from Science Question Answering

TLDR
A new dataset and model for textual entailment, derived from treating multiple-choice question-answering as an entailment problem, is presented, and it is demonstrated that one can improve accuracy on SCITAIL by 5% using a new neural model that exploits linguistic structure.
...