• Corpus ID: 233289562

KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding

  title={KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding},
  author={Keyur Faldu and A. Sheth and Prashant Kikani and Hemang Akabari},
Contextualized entity representations learned by state-of-the-art deep learning models (BERT, GPT, T5, etc) leverage the attention mechanism to learn the data context. However, these models are still blind to leverage the knowledge context present in the knowledge graph. Knowledge context can be understood as semantics about entities, and their relationship with neighboring entities in knowledge graphs. We propose a novel and effective technique to infuse knowledge context from knowledge graphs… 

Figures and Tables from this paper

Knowledge-Intensive Language Understanding for Explainable AI
To understand and validate an AI system’s outcomes (such as classification, recommendations, predictions), that lead to developing trust in the AI system, it is necessary to involve explicit domain knowledge that humans understand and use.
KG-ZESHEL: Knowledge Graph-Enhanced Zero-Shot Entity Linking
KG-ZESHEL is presented, a knowledge graph-enhanced zero-shot entity linking approach, which extends an existing BERT-based zero- shot entity linking Approach with mention and entity auxiliary information, and shows that the proposed approach outperforms the related Bert-based state-of-the-art entity linking models.
MMTM: Multi-Tasking Multi-Decoder Transformer for Math Word Problems
A novel model MMTM that leverages multi-tasking and multi-decoder during pre-training that achieves better mathematical reasoning ability and generalisability, which is demonstrated by outperforming the best state of the art baseline models from Seq2Seq, GTS, and Graph2Tree with a relative improvement on an adversarial challenge dataset SVAMP.
Symbolic Data Augmentation for Assisted Neural Reasoning
It is demonstrated that SDAR can boost the performance of smaller models to a comparable degree of or even surpass larger models with a magnitude more parameters, and establish the state-of-the-art single model performance on OPENBOOKQA.
Development and application of a digital humanities research platform for biographies of Malaysian personalities
Purpose Digital humanities research platform for biographies of Malaysia personalities (DHRP-BMP) was collaboratively developed by the Research Center for Chinese Cultural Subjectivity in Taiwan,
Spatial Commonsense Graph for Object Localisation in Partial Scenes
The proposed solution is based on a novel scene graph model, the Spatial Commonsense Graph (SCG), where objects are the nodes and edges define pairwise distances between them, enriched by concept nodes and relationships from a commonsense knowledge base.
Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems
This work inspects non-neural and neural methods to solve math word problems narrated in a natural language, and highlights the ability of these methods to be generalizable, mathematically reasonable, interpretable, and explainable.
Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings
The largest and most fine-grained explainable NLI crowdsourcing study to date reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.
Structure Inducing Pre-Training
We present a theoretical analysis from first principles that establishes a novel connection between relational inductive bias of pre-training and fine-tuning performance while providing an extended


K-BERT: Enabling Language Representation with Knowledge Graph
This work proposes a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge, which significantly outperforms BERT and reveals promising results in twelve NLP tasks.
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
A unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs is proposed.
ERNIE: Enhanced Language Representation with Informative Entities
This paper utilizes both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE) which can take full advantage of lexical, syntactic, and knowledge information simultaneously, and is comparable with the state-of-the-art model BERT on other common NLP tasks.
Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
Experimental results demonstrate that pre-training models using the proposed approach followed by fine-tuning achieve significant improvements over previous state-of-the-art models on two commonsense-related benchmarks, including CommonsenseQA and Winograd Schema Challenge.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
K-Adapter is proposed, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion and captures richer factual and commonsense knowledge than RoBERTa.
Convolutional 2D Knowledge Graph Embeddings
ConvE, a multi-layer convolutional network model for link prediction, is introduced, and it is found that ConvE achieves state-of-the-art Mean Reciprocal Rank across all datasets.
Semantics of the Black-Box: Can Knowledge Graphs Help Make Deep Learning Systems More Interpretable and Explainable?
This article demonstrates how knowledge, provided as a knowledge graph, is incorporated into DL using K- iL, and discusses the utility of K-iL towards interpretability and explainability.
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
A new version of the linked open data resource ConceptNet is presented that is particularly well suited to be used with modern NLP techniques such as word embeddings, with state-of-the-art results on intrinsic evaluations of word relatedness that translate into improvements on applications of word vectors, including solving SAT-style analogies.
Explainable AI Using Knowledge Graphs
This tutorial discusses the notion of explainability and interpretability through the use of knowledge graphs in Healthcare on the Web, Education Technology and details of knowledge-infused learning algorithms and its contribution to explainability for the above two applications that can be applied to any other domain using knowledge graphs.