Corpus ID: 220525491

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

@inproceedings{Chi2021InfoXLMAI,
  title={InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training},
  author={Zewen Chi and Li Dong and Furu Wei and Nan Yang and Saksham Singhal and Wenhui Wang and Xia Song and Xian-Ling Mao and He-yan Huang and M. Zhou},
  booktitle={NAACL},
  year={2021}
}
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same… Expand
Can Monolingual Pretrained Models Help Cross-Lingual Classification?
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation
Globetrotter: Unsupervised Multilingual Translation from Visual Alignment
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application
Rehoboam at the NTCIR-15 SHINRA2020-ML Task
A Survey on Contrastive Self-supervised Learning
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 45 REFERENCES
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Cross-Lingual Natural Language Generation via Pre-Training
Alternating Language Modeling for Cross-Lingual Pre-Training
Cross-lingual Language Model Pretraining
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
XNLI: Evaluating Cross-lingual Sentence Representations
Unsupervised Cross-lingual Representation Learning at Scale
...
1
2
3
4
5
...