SciBERT: A Pretrained Language Model for Scientific Text

@inproceedings{Beltagy2019SciBERTAP,
  title={SciBERT: A Pretrained Language Model for Scientific Text},
  author={Iz Beltagy and Kyle Lo and Arman Cohan},
  booktitle={EMNLP},
  year={2019}
}
Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et. al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence… Expand
Low-Resource Adaptation of Neural NLP Models
NLP-PINGAN-TECH @ CL-SciSumm 2020
Predictions For Pre-training Language Models
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 33 REFERENCES
Universal Language Model Fine-tuning for Text Classification
Publicly Available Clinical BERT Embeddings
Improving Language Understanding by Generative Pre-Training
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Deep contextualized word representations
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
...
1
2
3
4
...