SpanBERT: Improving Pre-training by Representing and Predicting Spans

@article{Joshi2019SpanBERTIP,
  title={SpanBERT: Improving Pre-training by Representing and Predicting Spans},
  author={Mandar Joshi and Danqi Chen and Yinhan Liu and Daniel S. Weld and Luke Zettlemoyer and Omer Levy},
  journal={Transactions of the Association for Computational Linguistics},
  year={2019},
  volume={8},
  pages={64-77}
}
  • Mandar Joshi, Danqi Chen, +3 authors Omer Levy
  • Published 2019
  • Computer Science
  • Transactions of the Association for Computational Linguistics
  • We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as… CONTINUE READING

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 144 CITATIONS

    FILTER CITATIONS BY YEAR

    2019
    2020

    CITATION STATISTICS

    • 28 Highly Influenced Citations

    • Averaged 72 Citations per year from 2019 through 2020

    • 200% Increase in citations per year in 2020 over 2019

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 69 REFERENCES