Publicly Available Clinical BERT Embeddings

@article{Alsentzer2019PubliclyAC,
  title={Publicly Available Clinical BERT Embeddings},
  author={Emily Alsentzer and J. Murphy and Willie Boag and W. Weng and Di Jin and Tristan Naumann and Matthew B. A. McDermott},
  journal={ArXiv},
  year={2019},
  volume={abs/1904.03323}
}
Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for… Expand
Enhancing Clinical Concept Extraction with Contextual Embedding
TLDR
The potential of contextual embeddings is demonstrated through the state-of-the-art performance these methods achieve on clinical concept extraction and the impact of the pretraining time of a large language model like ELMo or BERT is analyzed. Expand
EduBERT: Pretrained Deep Language Models for Learning Analytics
TLDR
It is demonstrated that using large pretrained models produces excellent results on common learning analytics tasks, and that a smaller, distilled version of the model produces the best results on two of the three tasks while limiting computational cost. Expand
Hurtful words: quantifying biases in clinical contextual word embeddings
TLDR
This work pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches to identify dangerous latent relationships that are captured by the contextual word embeddings. Expand
BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition
TLDR
The transfer learning process enhanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model, demonstrating that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. Expand
BertMCN: Mapping colloquial phrases to standard medical concepts using BERT and highway network
  • Katikapalli Subramanyam Kalyan, S. Sangeetha
  • Medicine, Computer Science
  • Artif. Intell. Medicine
  • 2021
TLDR
BERT, a pre-trained context sensitive deep language representation model advanced state-of-the-art performance in many NLP tasks and gating mechanism in highway layer helps the model to choose only important information. Expand
An Empirical Investigation towards Efficient Multi-Domain Language Model Pre-training
TLDR
An empirical investigation into known methods to mitigate catastrophic forgetting is conducted and it is found that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. Expand
Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction
TLDR
Inspired by BERT, Med-BERT is a contextualized embedding model pretrained on a structured EHR dataset of 28,490,650 patients that substantially improves the prediction accuracy and can boost the area under the receiver operating characteristics curve (AUC) by 1.21–6.14% in two disease prediction tasks from two clinical databases. Expand
Integrating Text Embedding with Traditional NLP Features for Clinical Relation Extraction
TLDR
This paper investigates whether traditional NLP features can be combined with word and sentence embeddings to improve relation extraction and develops new models that significantly outperformed all the baselines on the same dataset. Expand
Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base
TLDR
This paper shows that in three different downstream clinical NLP tasks, the pre-trained language model outperforms the corresponding model with no knowledge base information and other state-of-the-art models. Expand
UW-BHI at MEDIQA 2019: An Analysis of Representation Methods for Medical Natural Language Inference
TLDR
This paper compares the performance and internal representation of an Enhanced Sequential Inference Model (ESIM) between three experimental conditions based on the representation method: Bidirectional Encoder Representations from Transformers (BERT), Embeddings of Semantic Predications (ESP), or Cui2Vec. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 23 REFERENCES
Enhancing Clinical Concept Extraction with Contextual Embedding
TLDR
The potential of contextual embeddings is demonstrated through the state-of-the-art performance these methods achieve on clinical concept extraction and the impact of the pretraining time of a large language model like ELMo or BERT is analyzed. Expand
Lessons from Natural Language Inference in the Clinical Domain
TLDR
This work introduces MedNLI - a dataset annotated by doctors, performing a natural language inference task (NLI), grounded in the medical history of patients, and presents strategies to leverage transfer learning using datasets from the open domain and incorporate domain knowledge from external data and lexical sources. Expand
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TLDR
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks. Expand
A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
TLDR
This paper presents a deep learning architecture that builds on the latest NLP advances by incorporating deep contextualized word embeddings and variational drop out Bi-LSTMs and shows that the architecture achieves state-of-the-art performance on both data sets while also converging faster than other systems without the use of dictionaries or other knowledge sources. Expand
Clinical Concept Extraction with Contextual Word Embedding
TLDR
The proposed model achieved the best performance among reported baseline models and outperformed the state-of-the-art models by 3.4% in terms of F1-score. Expand
Deep Contextualized Word Representations
TLDR
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals. Expand
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
TLDR
It is shown how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Expand
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
TLDR
ScispaCy, a new Python library and models for practical biomedical/scientific text processing, which heavily leverages the spaCy library is described, which detail the performance of two packages of models released in scispa Cy and demonstrate their robustness on several tasks and datasets. Expand
Universal Language Model Fine-tuning for Text Classification
TLDR
This work proposes Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduces techniques that are key for fine- Tuning a language model. Expand
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
TLDR
This article introduces BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora that largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre- trained on biomedical Corpora. Expand
...
1
2
3
...