Language Chameleon: Transformation analysis between languages using Cross-lingual Post-training based on Pre-trained language models
@article{Son2022LanguageCT, title={Language Chameleon: Transformation analysis between languages using Cross-lingual Post-training based on Pre-trained language models}, author={Suhyune Son and Chanjun Park and Jungseob Lee and Midan Shim and Chanhee Lee and Yoonna Jang and Jaehyung Seo and Heu-Jeoung Lim}, journal={ArXiv}, year={2022}, volume={abs/2209.06422} }
As pre-trained language models become more resource-demanding, the inequality between resource-rich languages such as English and resource-scarce languages is worsening. This can be attributed to the fact that the amount of available training data in each language fol-lows the power-law distribution, and most of the languages belong to the long tail of the distribution. Some research areas attempt to mitigate this problem. For example, in cross-lingual transfer learning and multilingual…
Figures and Tables from this paper
References
SHOWING 1-10 OF 44 REFERENCES
Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models
- Computer Science, LinguisticsApplied Sciences
- 2021
Quantitative results from intrinsic and extrinsic evaluations show that the novel cross-lingual post-training approach outperforms several massively multilingual and monolingual pretrained language models in most settings and improves the data efficiency by a factor of up to 32 compared tomonolingual training.
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
- Computer Science, LinguisticsEMNLP
- 2019
It is found that doing fine-tuning on multiple languages together can bring further improvement in Unicoder, a universal language encoder that is insensitive to different languages.
Cross-lingual Language Model Pretraining
- Computer ScienceNeurIPS
- 2019
This work proposes two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingsual language model objective.
Unsupervised Cross-lingual Representation Learning at Scale
- Computer ScienceACL
- 2020
It is shown that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks, and the possibility of multilingual modeling without sacrificing per-language performance is shown for the first time.
On the Cross-lingual Transferability of Monolingual Representations
- Linguistics, Computer ScienceACL
- 2020
This work designs an alternative approach that transfers a monolingual model to new languages at the lexical level and shows that it is competitive with multilingual BERT on standard cross-lingUAL classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD).
Adapting Monolingual Models: Data can be Scarce when Language Similarity is High
- Computer ScienceFINDINGS
- 2021
This work retrain the lexical layers of four BERT-based models using data from two low-resource target language varieties, while the Transformer layers are independently fine-tuned on a POS-tagging task in the model's source language.
Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Computer Science, LinguisticsICLR
- 2020
A comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability, finding that the lexical overlap between languages plays a negligible role, while the depth of the network is an integral part of it.
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer
- Computer Science, LinguisticsEMNLP
- 2020
MAD-X is proposed, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations and introduces a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new language.
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models
- Computer ScienceACL
- 2022
A multilingual language model with 24 languages with entity representations is trained and it is shown that the model consistently outperforms word-based pretrained models in various cross-lingual transfer tasks.
On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing
- Computer Science, LinguisticsNAACL
- 2019
Investigating crosslingual transfer and posit that an orderagnostic model will perform better when transferring to distant foreign languages shows that RNN-based architectures transfer well to languages that are close to English, while self-attentive models have better overall cross-lingualtransferability and perform especially well on distant languages.