Representations of language in a model of visually grounded speech signal

@inproceedings{Chrupala2017RepresentationsOL,
  title={Representations of language in a model of visually grounded speech signal},
  author={Grzegorz Chrupala and Lieke Gelderloos and Afra Alishahi},
  booktitle={ACL},
  year={2017}
}
We present a visually grounded model of speech perception which projects spoken utterances and images to a joint semantic space. We use a multi-layer recurrent highway network to model the temporal nature of spoken speech, and show that it learns to extract both form and meaningbased linguistic knowledge from the input signal. We carry out an in-depth analysis of the representations used by different components of the trained model and show that encoding of semantic aspects tends to become… CONTINUE READING
Highly Cited
This paper has 31 citations. REVIEW CITATIONS
Related Discussions
This paper has been referenced on Twitter 65 times. VIEW TWEETS

From This Paper

Figures, tables, and topics from this paper.

Citations

Publications citing this paper.
Showing 1-10 of 22 extracted citations

Linguistic Unit Discovery from Multi-Modal Inputs in Unwritten Languages: Summary of the “Speaking Rosetta” JSALT 2017 Workshop

2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) • 2018
View 2 Excerpts

Object Referring in Visual Scene with Spoken Language

2018 IEEE Winter Conference on Applications of Computer Vision (WACV) • 2018

References

Publications referenced by this paper.
Showing 1-10 of 32 references

Deep multimodal semantic embeddings for speech and images

2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) • 2015
View 7 Excerpts
Highly Influenced

Adam: A Method for Stochastic Optimization

View 3 Excerpts
Highly Influenced

Memory visualization for gated recurrent neural networks in speech recognition

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) • 2017
View 2 Excerpts

Synthetically spoken COCO

Grzegorz Chrupała, Lieke Gelderloos, Afra Alishahi.
https://doi.org/10.5281/zenodo.400926. • 2017
View 1 Excerpt

Similar Papers

Loading similar papers…