Skip to Main content Skip to Navigation
Journal articles

Weakly Supervised Representation Learning for Audio-Visual Scene Analysis

Abstract : Audiovisual (AV) representation learning is an important task from the perspective of designing machines with the ability to understand complex events. To this end, we propose a novel multimodal framework that instantiates multiple instance learning. Specifically, we develop methods that identify events and localize corresponding AV cues in unconstrained videos. Importantly, this is done using weak labels where only video-level event labels are known without any information about their location in time. We show that the learnt representations are useful for performing several tasks such as event/object classification, audio event detection, audio source separation and visual object localization. An important feature of our method is its capacity to learn from unsynchronized audiovisual events. We also demonstrate our framework's ability to separate out the audio source of interest through a novel use of nonnegative matrix factorization. State-of-the-art classification results, with a F1-score of 65.0, are achieved on DCASE 2017 smart cars challenge data with promising generalization to diverse object types such as musical instruments. Visualizations of localized visual regions and audio segments substantiate our system's efficacy, especially when dealing with noisy situations where modality-specific cues appear asynchronously.
Document type :
Journal articles
Complete list of metadata

Cited literature [78 references]  Display  Hide  Download
Contributor : Gaël RICHARD Connect in order to contact the contributor
Submitted on : Monday, December 9, 2019 - 12:28:33 PM
Last modification on : Wednesday, November 3, 2021 - 6:18:09 AM
Long-term archiving on: : Tuesday, March 10, 2020 - 5:32:44 PM


Files produced by the author(s)


  • HAL Id : hal-02399993, version 1



Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc Q. K. Duong, Patrick Pérez, et al.. Weakly Supervised Representation Learning for Audio-Visual Scene Analysis. IEEE/ACM Transactions on Audio, Speech and Language Processing, Institute of Electrical and Electronics Engineers, 2019. ⟨hal-02399993⟩



Record views


Files downloads