ERNIE: Enhanced Language Representation with Informative Entities

@inproceedings{Zhang2019ERNIEEL,
  title={ERNIE: Enhanced Language Representation with Informative Entities},
  author={Zhengyan Zhang and Xu Han and Zhiyuan Liu and Xin Jiang and Maosong Sun and Qun Liu},
  booktitle={ACL},
  year={2019}
}
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. [...] Key Method In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE…Expand
CKG: Dynamic Representation Based on Context and Knowledge Graph
TLDR
This paper argues that entities in KGs could be used to enhance the correct semantic meaning of language sentences and proposes a new method CKG: Dynamic Representation Based on Context and Knowledge Graph, which can extract rich semantic information of large corpus and make full use of inside information such as co-occurrence in large corpus. Expand
K-BERT: Enabling Language Representation with Knowledge Graph
TLDR
This work proposes a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge, which significantly outperforms BERT and demonstrates that K-berT is an excellent choice for solving the knowledge-driven problems that require experts. Expand
Enriching Language Models with Semantics
TLDR
This paper highlights existing approaches to combine different types of semantics with language models during the pre-training or fine-tuning phase to improve various natural language processing tasks. Expand
ERNIE-NLI: Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI
TLDR
Using the ERNIE architecture, a detailed analysis is provided on the types of knowledge that result in a performance increase on the Natural Language Inference task, specifically on the Multi-Genre Natural language Inference Corpus (MNLI). Expand
Learning Better Sentence Representation with Syntax Information
TLDR
A novel approach to combining syntax information with a pre-trained language model using a dependency syntax expansion (DSE) model, which achieves 91.2% accuracy, outperforming the baseline model by 37.8% on sentence completion task and gets competitive performance by 75.1% F1 score on relation extraction task. Expand
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models
TLDR
This study trains a multilingual language model with 24 languages with entity representations and shows the model consistently outperforms word-based pretrained models in various crosslingual transfer tasks. Expand
HORNET: Enriching Pre-trained Language Representations with Heterogeneous Knowledge Sources
  • Taolin Zhang, Zerui Cai, +6 authors Jun Huang
  • Computer Science
  • CIKM
  • 2021
TLDR
A novel KEPLM named HORNET is proposed, which integrates Heterogeneous knowledge from various structured and unstructured sources into the Roberta NETwork and hence takes full advantage of both linguistic and factual knowledge simultaneously, and design a hybrid attention heterogeneous graph convolution network (HaHGCN). Expand
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
TLDR
A unified framework named ERNIE 3.0 is proposed for pre-training large-scale knowledge enhanced models that fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few- shot learning or fine-tuning. Expand
K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining
TLDR
The experimental results show that solely by adding external knowledge to transformer can improve the learning performance on many NLP tasks. Expand
Language Models are Open Knowledge Graphs
TLDR
This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision, and proposes an unsupervised method to cast the knowledge contained within language models into KGs. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 63 REFERENCES
Neural Natural Language Inference Models Enhanced with External Knowledge
TLDR
This paper enrichs the state-of-the-art neural natural language inference models with external knowledge and demonstrates that the proposed models improve neural NLI models to achieve the state of theart performance on the SNLI and MultiNLI datasets. Expand
Neural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text
TLDR
A general joint representation learning framework for knowledge acquisition on two tasks, knowledge graph completion and relation extraction from text, where models trained under this joint framework are significantly improved in comparison with other baselines. Expand
ERNIE: Enhanced Representation through Knowledge Integration
TLDR
Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. Expand
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
TLDR
A benchmark of nine diverse NLU tasks, an auxiliary dataset for probing models for understanding of specific linguistic phenomena, and an online platform for evaluating and comparing models, which favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. Expand
Improving Language Understanding by Generative Pre-Training
TLDR
The general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. Expand
Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision
TLDR
This paper proposes a novel method for joint representation learning of cross-lingual words and entities that captures mutually complementary knowledge, and enables cross-lingsual inferences among knowledge bases and texts. Expand
Semi-supervised sequence tagging with bidirectional language models
TLDR
A general semi-supervised approach for adding pre- trained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers. Expand
Position-aware Attention and Supervised Data Improve Slot Filling
TLDR
An effective new model is proposed, which combines an LSTM sequence model with a form of entity position-aware attention that is better suited to relation extraction that builds TACRED, a large supervised relation extraction dataset obtained via crowdsourcing and targeted towards TAC KBP relations. Expand
Multi-level Representations for Fine-Grained Typing of Knowledge Base Entities
TLDR
Methods for learning multi-level representations of entities on three complementary levels are presented, confirming experimentally that each level of representation contributes complementary information and a joint representation of all three levels improves the existing embedding based baseline for fine-grained entity typing by a large margin. Expand
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TLDR
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks. Expand
...
1
2
3
4
5
...