Skip to Main content Skip to Navigation
Conference papers


Abstract : Lyrics-to-audio alignment methods have recently reported impressive results, opening the door to practical applications such as karaoke and within song navigation. However , most studies focus on a single language-usually En-glish-for which annotated data are abundant. The question of their ability to generalize to other languages, especially in low (or even zero) training resource scenarios has been so far left unexplored. In this paper, we address the lyrics-to-audio alignment task in a generalized multilingual setup. More precisely, this investigation presents the first (to the best of our knowledge) attempt to create a language-independent lyrics-to-audio alignment system. Building on a Recurrent Neural Network (RNN) model trained with a Connectionist Temporal Classification (CTC) algorithm, we study the relevance of different intermediate representations, either character or phoneme, along with several strategies to design a training set. The evaluation is conducted on multiple languages with a varying amount of data available, from plenty to zero. Results show that learning from diverse data and using a universal phoneme set as an intermediate representation yield the best generalization performances.
Complete list of metadata

Cited literature [34 references]  Display  Hide  Download
Contributor : Gaël Richard Connect in order to contact the contributor
Submitted on : Monday, November 9, 2020 - 7:19:22 PM
Last modification on : Tuesday, September 21, 2021 - 2:16:05 PM
Long-term archiving on: : Wednesday, February 10, 2021 - 7:53:36 PM


Publisher files allowed on an open archive


Distributed under a Creative Commons Attribution 4.0 International License


  • HAL Id : hal-02996940, version 1



Andrea Vaglio, Romain Hennequin, Manuel Moussallam, Gael Richard, Florence d'Alché-Buc. MULTILINGUAL LYRICS-TO-AUDIO ALIGNMENT. International Society for Music Information Retrieval Conference (ISMIR), Oct 2020, Montreal, Canada. ⟨hal-02996940⟩



Record views


Files downloads