Knowledge Enhanced Contextual Word Representations

@inproceedings{Peters2019KnowledgeEC,
  title={Knowledge Enhanced Contextual Word Representations},
  author={Matthew E. Peters and Mark Neumann and Robert L Logan IV and Roy Schwartz and Vidur Joshi and Sameer Singh and Noah A. Smith},
  booktitle={EMNLP},
  year={2019}
}
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. [...] Key Method For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention.Expand
DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections
TLDR
This paper radically expands the notion of context to include any available text related to an entity, enabling a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision. Expand
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
TLDR
The proposed model TOME is a Transformer that accesses the information through internal memory layers in which each entity mention in the input passage attends to the mention memory, which enables synthesis of and reasoning over many disparate sources of information within a single Transformer model. Expand
Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
TLDR
This work proposes two tasks -- an entity typing and a relation typing task -- that evaluate the performance of contextual and KB embeddings jointly at the same level and evaluated a concatenated model ofContextual and KBembeddings with these two tasks, obtaining conclusive results on the first task. Expand
Relational world knowledge representation in contextual language models: A review
TLDR
It is concluded that LMs and KBs are complementary representation tools, as KBs provide a high standard of factual precision which can in turn be flexibly and expressively modeled by LMs, and provide suggestions for future research in this direction. Expand
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
TLDR
New pretrained contextualized representations of words and entities based on the bidirectional transformer, and an entity-aware self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores are proposed. Expand
Injecting Knowledge Base Information into End-to-End Joint Entity and Relation Extraction and Coreference Resolution
TLDR
This work studies how to inject information from a knowledge base (KB) in a joint information extraction (IE) model, based on unsupervised entity linking, and reveals the advantage of using the attention-based approach. Expand
Simultaneously Self-Attending to Text and Entities for Knowledge-Informed Text Representations
Pre-trained language models have emerged as highly successful methods for learning good text representations. However, the amount of structured knowledge retained in such models, and how (if at all)Expand
Contextualized Representations Using Textual Encyclopedic Knowledge
TLDR
It is shown that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders and knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text. Expand
CoLAKE: Contextualized Language and Knowledge Embedding
TLDR
The Contextualized Language and Knowledge Embedding (CoLAKE) is proposed, which jointly learns contextualized representation for both language and knowledge with the extended MLM objective, and achieves surprisingly high performance on a synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language andknowledge representation. Expand
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity
TLDR
The experiments suggest that the standard BERT (LIBERT), specialized for the word-level semantic similarity, yields better performance than the lexically blind “vanilla” BERT on several language understanding tasks, and shows consistent gains on 3 benchmarks for lexical simplification. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 72 REFERENCES
Entity Linking via Joint Encoding of Types, Descriptions, and Context
TLDR
This work presents a neural, modular entity linking system that learns a unified dense representation for each entity using multiple sources of information, such as its description, contexts around its mentions, and its fine-grained types. Expand
End-to-End Neural Entity Linking
TLDR
This work proposes the first neural end-to-end EL system that jointly discovers and links entities in a text document and shows that it significantly outperforms popular systems on the Gerbil platform when enough training data is available. Expand
ERNIE: Enhanced Language Representation with Informative Entities
TLDR
This paper utilizes both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE) which can take full advantage of lexical, syntactic, and knowledge information simultaneously, and is comparable with the state-of-the-art model BERT on other common NLP tasks. Expand
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
TLDR
A new version of the linked open data resource ConceptNet is presented that is particularly well suited to be used with modern NLP techniques such as word embeddings, with state-of-the-art results on intrinsic evaluations of word relatedness that translate into improvements on applications of word vectors, including solving SAT-style analogies. Expand
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
TLDR
This work presents a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM, and suggests they could be useful in a wide variety of NLP tasks. Expand
Improving Relation Extraction by Pre-trained Language Representations
TLDR
TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. Expand
Knowledge Graph and Text Jointly Embedding
TLDR
Large scale experiments on Freebase and a Wikipedia/NY Times corpus show that jointly embedding brings promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text. Expand
Learning Entity and Relation Embeddings for Knowledge Graph Completion
TLDR
TransR is proposed to build entity and relation embeddings in separate entity space and relation spaces to build translations between projected entities and to evaluate the models on three tasks including link prediction, triple classification and relational fact extraction. Expand
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
TLDR
This work introduces the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context that enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. Expand
Robust Disambiguation of Named Entities in Text
TLDR
A robust method for collective disambiguation is presented, by harnessing context from knowledge bases and using a new form of coherence graph that significantly outperforms prior methods in terms of accuracy, with robust behavior across a variety of inputs. Expand
...
1
2
3
4
5
...