Knowledge Enhanced Contextual Word Representations

@inproceedings{Peters2019KnowledgeEC,
  title={Knowledge Enhanced Contextual Word Representations},
  author={Matthew E. Peters and Mark Neumann and IV RobertL.Logan and Roy Schwartz and Vidur Joshi and Sameer Singh and Noah A. Smith},
  booktitle={EMNLP},
  year={2019}
}
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual… 

Figures and Tables from this paper

DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections
TLDR
This paper radically expands the notion of context to include any available text related to an entity, enabling a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision.
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
TLDR
The proposed model - TOME - is a Transformer that accesses the information through internal memory layers in which each entity mention in the input passage attends to the mention memory, which enables synthesis of and reasoning over many disparate sources of information within a single Transformer model.
Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
TLDR
This work proposes two tasks -- an entity typing and a relation typing task -- that evaluate the performance of contextual and KB embeddings jointly at the same level and evaluated a concatenated model ofContextual and KBembeddings with these two tasks, obtaining conclusive results on the first task.
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
TLDR
New pretrained contextualized representations of words and entities based on the bidirectional transformer, and an entity-aware self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores are proposed.
Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations
TLDR
This paper aligns structured knowledge contained in temporal knowledge graphs with their textual descriptions extracted from news articles and proposes a novel knowledge-text prediction task to inject the abundant information from descriptions into temporal knowledge embeddings.
Relational World Knowledge Representation in Contextual Language Models: A Review
TLDR
This work proposes to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision, and provides a high-level, extensible taxonomy for knowledge representation in L Ms.
Contextual Graph Representation Learning in Text Disambiguation
  • M. Saeidi
  • Computer Science
    MLSMKG@PKDD/ECML
  • 2021
TLDR
ContextBERT is a novel approach to producing sense embeddings for the lexical meanings within a lexical knowledge base, using pre-trained BERT model and the novel difference in this representation is the integration of the knowledge base information and the input text.
Injecting Knowledge Base Information into End-to-End Joint Entity and Relation Extraction and Coreference Resolution
TLDR
This work studies how to inject information from a knowledge base (KB) in a joint information extraction (IE) model, based on unsupervised entity linking, and reveals the advantage of using the attention-based approach.
Simultaneously Self-Attending to Text and Entities for Knowledge-Informed Text Representations
TLDR
This work aims at directly learning text representations which leverage structured knowledge about entities mentioned in the text which can be particularly beneficial for downstream tasks which are knowledge-intensive.
Contextualized Representations Using Textual Encyclopedic Knowledge
TLDR
It is shown that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders and knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 73 REFERENCES
Entity Linking via Joint Encoding of Types, Descriptions, and Context
TLDR
This work presents a neural, modular entity linking system that learns a unified dense representation for each entity using multiple sources of information, such as its description, contexts around its mentions, and its fine-grained types.
End-to-End Neural Entity Linking
TLDR
This work proposes the first neural end-to-end EL system that jointly discovers and links entities in a text document and shows that it significantly outperforms popular systems on the Gerbil platform when enough training data is available.
ERNIE: Enhanced Language Representation with Informative Entities
TLDR
This paper utilizes both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE) which can take full advantage of lexical, syntactic, and knowledge information simultaneously, and is comparable with the state-of-the-art model BERT on other common NLP tasks.
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
TLDR
A new version of the linked open data resource ConceptNet is presented that is particularly well suited to be used with modern NLP techniques such as word embeddings, with state-of-the-art results on intrinsic evaluations of word relatedness that translate into improvements on applications of word vectors, including solving SAT-style analogies.
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
TLDR
This work presents a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM, and suggests they could be useful in a wide variety of NLP tasks.
Improving Relation Extraction by Pre-trained Language Representations
TLDR
TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions.
Knowledge Graph and Text Jointly Embedding
TLDR
Large scale experiments on Freebase and a Wikipedia/NY Times corpus show that jointly embedding brings promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text.
Learning Entity and Relation Embeddings for Knowledge Graph Completion
TLDR
TransR is proposed to build entity and relation embeddings in separate entity space and relation spaces to build translations between projected entities and to evaluate the models on three tasks including link prediction, triple classification and relational fact extraction.
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
TLDR
This work introduces the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context that enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens.
Robust Disambiguation of Named Entities in Text
TLDR
A robust method for collective disambiguation is presented, by harnessing context from knowledge bases and using a new form of coherence graph that significantly outperforms prior methods in terms of accuracy, with robust behavior across a variety of inputs.
...
1
2
3
4
5
...