Refining Targeted Syntactic Evaluation of Language Models

@inproceedings{Newman2021RefiningTS,
  title={Refining Targeted Syntactic Evaluation of Language Models},
  author={Benjamin A. Newman and Kai-Siang Ang and Julia Gong and John Hewitt},
  booktitle={NAACL},
  year={2021}
}
Targeted syntactic evaluation of subject-verb number agreement in English (TSE) evaluates language models’ syntactic knowledge using hand-crafted minimal pairs of sentences that differ only in the main verb’s conjugation. The method evaluates whether language models rate each grammatical sentence as more likely than its ungrammatical counterpart. We identify two distinct goals for TSE. First, evaluating the systematicity of a language model’s syntactic knowledge: given a sentence, can it… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 29 REFERENCES
Targeted Syntactic Evaluation of Language Models
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Distinct patterns of syntactic agreement errors in recurrent networks and humans
...
1
2
3
...