Learning Bimodal Structure in Audio–Visual Data

G. Monaci; P. Vandergheynst; F. Sommer

DOI:10.1109/TNN.2009.2032182

Corpus ID: 11580747

Learning Bimodal Structure in Audio–Visual Data

@article{Monaci2009LearningBS,
  title={Learning Bimodal Structure in Audio–Visual Data},
  author={G. Monaci and P. Vandergheynst and F. Sommer},
  journal={IEEE Transactions on Neural Networks},
  year={2009},
  volume={20},
  pages={1898-1910}
}

G. Monaci, P. Vandergheynst, F. Sommer

Published 2009

Computer Science, Medicine

IEEE Transactions on Neural Networks

A novel model is presented to learn bimodally informative structures from audio-visual signals. The signal is represented as a sparse sum of audio-visual kernels. Each kernel is a bimodal function consisting of synchronous snippets of an audio waveform and a spatio-temporal visual basis function. To represent an audio-visual signal, the kernels can be positioned independently and arbitrarily in space and time. The proposed algorithm uses unsupervised learning to form dictionaries of bimodal… CONTINUE READING

View on IEEE

Share This Paper

34 Citations

Highly Influencial Citations

Background Citations

Methods Citations

Figures, Tables, and Topics from this paper

Figures and Tables

Reverberant speech separation based on audio-visual dictionary learning and binaural cues

Qingju Liu, W. Wang, P. Jackson, M. Barnard
Computer Science
2012 IEEE Statistical Signal Processing Workshop (SSP)
2012

Source Separation of Convolutive and Noisy Mixtures Using Audio-Visual Dictionary Learning and Probabilistic Time-Frequency Masking

Qingju Liu, W. Wang, P. Jackson, M. Barnard, J. Kittler, J. Chambers
Computer Science
IEEE Transactions on Signal Processing
2013

24
Highly Influenced
PDF

Sequential Audio-Visual Correspondence With Alternating Diffusion Kernels

D. Dov, R. Talmon, I. Cohen
Computer Science
IEEE Transactions on Signal Processing
2018

Audio visual speech source separation via improved context dependent association model

Alireza Kazemi, R. Boostani, F. Sobhanmanesh
Computer Science
EURASIP J. Adv. Signal Process.
2014

Lip movement and speech synchronization detection based on multimodal shift-invariant dictionary

Zhengyu Zhu, Xiaohui Feng, J. Yang
Computer Science
2015 IEEE 16th International Conference on Communication Technology (ICCT)
2015

Audiovisual Speech Source Separation

B. Rivet, W. Wang, Mohsen Naqvi, J. Chambers
Computer Science
2014

2
Highly Influenced

Use of bimodal coherence to resolve the permutation problem in convolutive BSS

Qingju Liu, W. Wang, P. Jackson
Computer Science
Signal Process.
2012

Robust front-end for audio, visual and audio–visual speech classification

L. Terissi, Gonzalo D. Sad, J. C. Gómez
Computer Science
Int. J. Speech Technol.
2018

Audiovisual Speech Source Separation: An overview of key methodologies

B. Rivet, W. Wang, S. Naqvi, J. Chambers
Computer Science
IEEE Signal Processing Magazine
2014

Audio-visual localization with hierarchical topographic maps: Modeling the superior colliculus

Matthew C. Casey, Athanasios Pavlou, A. Timotheou
Computer Science
Neurocomputing
2012

References

SHOWING 1-10 OF 72 REFERENCES

SORT BY

Audiovisual Gestalts

G. Monaci, P. Vandergheynst
Computer Science
2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06)
2006

Audio Vision: Using Audio-Visual Synchrony to Locate Sounds

J. Hershey, J. Movellan
Computer Science
NIPS
1999

Analysis of multimodal sequences using geometric video representations

G. Monaci, Òscar Divorra Escoda, P. Vandergheynst
Computer Science
Signal Process.
2006

Video assisted speech source separation

W. Wang, D. Cosker, Y. Hicks, S. Sanei, J. Chambers
Computer Science
Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005.
2005