Probing Natural Language Inference Models through Semantic Fragments

  title={Probing Natural Language Inference Models through Semantic Fragments},
  author={Kyle Richardson and H. Hu and L. Moss and A. Sabharwal},
  • Kyle Richardson, H. Hu, +1 author A. Sabharwal
  • Published in AAAI 2020
  • Computer Science
  • Do state-of-the-art models for language understanding already have, or can they easily learn, abilities such as boolean coordination, quantification, conditionals, comparatives, and monotonicity reasoning (i.e., reasoning about word substitutions in sentential contexts. [...] Key Result Our experiments, using a library of 8 such semantic fragments, reveal two remarkable findings: (a) State-of-the-art models, including BERT, that are pre-trained on existing NLI benchmark datasets perform poorly on these new…Expand Abstract
    28 Citations

    Figures, Tables, and Topics from this paper.

    Logical Inferences with Comparatives and Generalized Quantifiers
    Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
    • 1
    • PDF
    How Context Affects Language Models' Factual Predictions
    • 10
    • PDF
    Transformers as Soft Reasoners over Language
    • 19
    • PDF
    Probing Linguistic Systematicity
    • 5
    • PDF
    Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
    • 6
    • Highly Influenced
    • PDF


    A logical-based corpus for cross-lingual evaluation
    • 3
    • PDF
    A large annotated corpus for learning natural language inference
    • 1,558
    • PDF
    Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
    • 33
    • PDF
    Enhanced LSTM for Natural Language Inference
    • 512
    • Highly Influential
    • PDF
    Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
    • 176
    • PDF
    Stress Test Evaluation for Natural Language Inference
    • 97
    • Highly Influential
    • PDF
    Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
    • 13
    • Highly Influential
    • PDF