BERT is Not an Interlingua and the Bias of Tokenization

@inproceedings{Singh2019BERTIN,
  title={BERT is Not an Interlingua and the Bias of Tokenization},
  author={J. Singh and Bryan McCann and R. Socher and Caiming Xiong},
  booktitle={DeepLo@EMNLP-IJCNLP},
  year={2019}
}
Multilingual transfer learning can benefit both high- and low-resource languages, but the source of these improvements is not well understood. Cananical Correlation Analysis (CCA) of the internal representations of a pre- trained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space. This effect is magnified at deeper layers, suggesting that the model does not progressively abstract semantic con- tent… Expand
23 Citations
Finding Universal Grammatical Relations in Multilingual BERT
  • 24
  • Highly Influenced
  • PDF
It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT
  • 2
  • Highly Influenced
  • PDF
Probing Pretrained Language Models for Lexical Semantics
  • 13
  • Highly Influenced
  • PDF
Identifying Necessary Elements for BERT's Multilinguality
  • 4
  • PDF
Identifying Elements Essential for BERT’s Multilinguality
  • 5
  • PDF
...
1
2
3
...

References

SHOWING 1-10 OF 31 REFERENCES
Investigating Multilingual NMT Representations at Scale
  • 34
  • Highly Influential
  • PDF
Linguistic Knowledge and Transferability of Contextual Representations
  • 265
  • PDF
Multilingual Models for Compositional Distributed Semantics
  • 285
  • PDF
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
  • 335
  • PDF
Cross-lingual Language Model Pretraining
  • 830
  • PDF
Inducing Crosslingual Distributed Representations of Words
  • 330
  • PDF
Understanding Learning Dynamics Of Language Models with SVCCA
  • 38
  • PDF
Bilingual Word Representations with Monolingual Quality in Mind
  • 280
  • PDF
Word Translation Without Parallel Data
  • 846
  • PDF
Deep contextualized word representations
  • 5,561
  • PDF
...
1
2
3
4
...