Corpus ID: 236318351

Modelling Latent Translations for Cross-Lingual Transfer

  title={Modelling Latent Translations for Cross-Lingual Transfer},
  author={E. Ponti and Julia Kreutzer and Ivan Vulic and Siva Reddy},
While achieving state-of-the-art results in multiple tasks and languages, translation-based cross-lingual transfer is often overlooked in favour of massively multilingual pre-trained encoders. Arguably, this is due to its main limitations: 1) translation errors percolating to the classification phase and 2) the insufficient expressiveness of the maximum-likelihood translation. To remedy this, we propose a new technique that integrates both steps of the traditional pipeline (translation and… Expand

Figures and Tables from this paper

Visually Grounded Reasoning across Languages and Cultures
A new protocol is devised to construct an ImageNet-style hierarchy representative of more languages and cultures, and a multilingual dataset for Multicultural Reasoning over Vision and Language (MaRVL) is created by eliciting statements from native speaker annotators about pairs of images. Expand


XNLI: Evaluating Cross-lingual Sentence Representations
This work constructs an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including low-resource languages such as Swahili and Urdu and finds that XNLI represents a practical and challenging evaluation suite and that directly translating the test data yields the best performance among available baselines. Expand
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. Expand
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
This paper evaluates the cross-lingual effectiveness of representations from the encoder of a massively multilingual NMT model on 5 downstream classification and sequence labeling tasks covering a diverse set of over 50 languages and shows gains in zero-shot transfer in 4 out of these 5 tasks. Expand
Improving Neural Machine Translation Models with Monolingual Data
This work pairs monolingual training data with an automatic back-translation, and can treat it as additional parallel training data, and obtains substantial improvements on the WMT 15 task English German, and for the low-resourced IWSLT 14 task Turkish->English. Expand
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
It is demonstrated that the inexpensive few-shot transfer (i.e., additional fine-tuning on a few target-language instances) is surprisingly effective across the board, warranting more research efforts reaching beyond the limiting zero-shot conditions. Expand
Unsupervised Cross-lingual Representation Learning at Scale
It is shown that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks, and the possibility of multilingual modeling without sacrificing per-language performance is shown for the first time. Expand
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
GNMT, Google's Neural Machine Translation system, is presented, which attempts to address many of the weaknesses of conventional phrase-based translation systems and provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delicited models. Expand
Dual Learning for Machine Translation
Experiments show that dual-NMT works very well on English ↔ French translation; especially, by learning from monolingual data, it achieves a comparable accuracy to NMT trained from the full bilingual data for the French-to-English translation task. Expand
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
This work shows that multilingual translation models can be created through multilingual finetuning, and demonstrates that pretrained models can been extended to incorporate additional languages without loss of performance. Expand
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity
An alternative reward function for optimizing NMT systems that is based on recent work in semantic similarity is introduced, and it is found that training with the proposed metric results in better translations as evaluated by BLEU, semantic similarity, and human evaluation, and that the optimization procedure converges faster. Expand