Corpus ID: 85518318

SciBERT: Pretrained Contextualized Embeddings for Scientific Text

@article{Beltagy2019SciBERTPC,
  title={SciBERT: Pretrained Contextualized Embeddings for Scientific Text},
  author={Iz Beltagy and Arman Cohan and Kyle Lo},
  journal={ArXiv},
  year={2019},
  volume={abs/1903.10676}
}
  • Iz Beltagy, Arman Cohan, Kyle Lo
  • Published 2019
  • Computer Science
  • ArXiv
  • Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained contextualized embedding model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging… CONTINUE READING

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 91 CITATIONS, ESTIMATED 98% COVERAGE

    BioFLAIR: Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks

    VIEW 4 EXCERPTS
    CITES BACKGROUND, METHODS & RESULTS
    HIGHLY INFLUENCED

    Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

    VIEW 2 EXCERPTS
    CITES BACKGROUND & METHODS

    Keyphrase Extraction as Sequence Labeling Using Contextualized Embeddings

    VIEW 10 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    Pre-Training BERT on Domain Resources for Short Answer Grading

    VIEW 2 EXCERPTS
    CITES BACKGROUND

    FILTER CITATIONS BY YEAR

    2019
    2020

    CITATION STATISTICS

    • 25 Highly Influenced Citations

    • Averaged 45 Citations per year from 2019 through 2020

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 30 REFERENCES

    Deep contextualized word representations

    VIEW 3 EXCERPTS