Lifelong Domain Word Embedding via Meta-Learning

@inproceedings{Xu2018LifelongDW,
  title={Lifelong Domain Word Embedding via Meta-Learning},
  author={Hu Xu and B. Liu and Lei Shu and Philip S. Yu},
  booktitle={IJCAI},
  year={2018}
}
Learning high-quality domain word embeddings is important for achieving good performance in many NLP tasks. General-purpose embeddings trained on large-scale corpora are often sub-optimal for domain-specific applications. However, domain-specific tasks often do not have large in-domain corpora for training high-quality domain embeddings. In this paper, we propose a novel lifelong learning setting for domain embedding. That is, when performing the new domain embedding, the system has seen many… 

Figures and Tables from this paper

Lifelong Learning of Topics and Domain-Specific Word Embeddings
TLDR
This study develops a lifelong collaborative model (LCM) based on non-negative matrix factorization to accurately learn topics and domain-specific word embeddings to overcome data diversity of the in-domain corpus.
Fusing Vector Space Models for Domain-Specific Applications
TLDR
This work proposes a new method that automatically combines multiple domain-specific embeddings, selected from a wide range of pre-trained domain- specific embedDings, to improve their combined expressive power.
Inferring Multilingual Domain-Specific Word Embeddings From Large Document Corpora
TLDR
A new methodology to automatically infer aligned domain-specific word embeddings for a target language on the basis of the general-purpose and domain- specific models available for a source language (typically, English) is proposed.
TransDrift: Modeling Word-Embedding Drift using Transformer
TLDR
This work proposes TransDrift, a transformer-based prediction model for word embeddings that accurately learns the dynamics of the embedding drift and predicts the future embedding, and makes significantly more accurate predictions than the baselines.
Continual Lifelong Learning in Natural Language Processing: A Survey
TLDR
This work looks at the problem of CL through the lens of various NLP tasks, and discusses major challenges in CL and current methods applied in neural network models.
Lifelong Language Knowledge Distillation
TLDR
The proposed Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation, is presented.
Induction Networks for Few-Shot Text Classification
TLDR
This paper proposes a novel Induction Network to learn a generalized class-wise representation of each class in the support set, by innovatively leveraging the dynamic routing algorithm in meta-learning and finds the model is able to induce and generalize better.
...
...

References

SHOWING 1-10 OF 48 REFERENCES
A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings
TLDR
This paper presents a simple yet effective method for learning word embeddings based on text from different domains and demonstrates the effectiveness of the approach through extensive experiments on various down-stream NLP tasks.
Think Globally, Embed Locally - Locally Linear Meta-embedding of Words
TLDR
Experimental results on semantic similarity, word analogy, relation classification, and short-text classification tasks show that the proposed unsupervised locally linear meta-embedding learning method to significantly outperform prior methods in several benchmark datasets, establishing a new state of the art for meta- embeddings.
Word Representations: A Simple and General Method for Semi-Supervised Learning
TLDR
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines.
Topic Modeling using Topics from Many Domains, Lifelong Learning and Big Data
TLDR
A novel method to mine prior knowledge dynamically in the modeling process, and then a new topic model to use the knowledge to guide the model inference, which offers a novel lifelong learning algorithm for topic discovery.
Learning Word Vectors for Sentiment Analysis
TLDR
This work presents a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term--document information as well as rich sentiment content, and finds it out-performs several previously introduced methods for sentiment classification.
Enriching Word Vectors with Subword Information
TLDR
A new approach based on the skipgram model, where each word is represented as a bag of character n-grams, with words being represented as the sum of these representations, which achieves state-of-the-art performance on word similarity and analogy tasks.
GloVe: Global Vectors for Word Representation
TLDR
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction
TLDR
This paper is the first to report such double embeddings based CNN model for aspect extraction and achieve very good results, outperforming state-of-the-art sophisticated existing methods.
Linguistic Regularities in Continuous Space Word Representations
TLDR
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
Lifelong-RL: Lifelong Relaxation Labeling for Separating Entities and Aspects in Opinion Targets
TLDR
This paper proposes a novel algorithm, called Lifelong-RL, to solve the problem of opinion mining based on lifelong machine learning and relaxation labeling, and extensive experiments show that the proposed algorithm Lifetime-RL outperforms baseline methods markedly.
...
...