Language Chameleon: Transformation analysis between languages using Cross-lingual Post-training based on Pre-trained language models
@article{Son2022LanguageCT, title={Language Chameleon: Transformation analysis between languages using Cross-lingual Post-training based on Pre-trained language models}, author={Suhyune Son and Chanjun Park and Jungseob Lee and Midan Shim and Chanhee Lee and Yoonna Jang and Jaehyung Seo and Heu-Jeoung Lim}, journal={ArXiv}, year={2022}, volume={abs/2209.06422} }
As pre-trained language models become more resource-demanding, the inequality between resource-rich languages such as English and resource-scarce languages is worsening. This can be attributed to the fact that the amount of available training data in each language fol-lows the power-law distribution, and most of the languages belong to the long tail of the distribution. Some research areas attempt to mitigate this problem. For example, in cross-lingual transfer learning and multilingual…
Figures and Tables from this paper
References
SHOWING 1-10 OF 44 REFERENCES
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
- Computer Science, LinguisticsEMNLP
- 2019
It is found that doing fine-tuning on multiple languages together can bring further improvement in Unicoder, a universal language encoder that is insensitive to different languages.
Cross-lingual Language Model Pretraining
- Computer ScienceNeurIPS
- 2019
This work proposes two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingsual language model objective.
Unsupervised Cross-lingual Representation Learning at Scale
- Computer ScienceACL
- 2020
It is shown that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks, and the possibility of multilingual modeling without sacrificing per-language performance is shown for the first time.
On the Cross-lingual Transferability of Monolingual Representations
- Linguistics, Computer ScienceACL
- 2020
This work designs an alternative approach that transfers a monolingual model to new languages at the lexical level and shows that it is competitive with multilingual BERT on standard cross-lingUAL classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD).
Adapting Monolingual Models: Data can be Scarce when Language Similarity is High
- Computer ScienceFINDINGS
- 2021
This work retrain the lexical layers of four BERT-based models using data from two low-resource target language varieties, while the Transformer layers are independently finetuned on a POS-tagging task in the model’s source language.
Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Computer Science, LinguisticsICLR
- 2020
A comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability, finding that the lexical overlap between languages plays a negligible role, while the depth of the network is an integral part of it.
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer
- Computer Science, LinguisticsEMNLP
- 2020
MAD-X is proposed, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations and introduces a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new language.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Computer ScienceJ. Mach. Learn. Res.
- 2020
This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.
On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing
- Computer Science, LinguisticsNAACL
- 2019
Investigating crosslingual transfer and posit that an orderagnostic model will perform better when transferring to distant foreign languages shows that RNN-based architectures transfer well to languages that are close to English, while self-attentive models have better overall cross-lingualtransferability and perform especially well on distant languages.
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Computer Science, LinguisticsICML
- 2020
The Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark is introduced, a multi-task benchmark for evaluating the cross-lingually generalization capabilities of multilingual representations across 40 languages and 9 tasks.