Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures - Equipe Signal, Statistique et Apprentissage Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures

Résumé

Communicative gestures and speech prosody are tightly linked. Our aim is to predict when gestures are performed based on prosody. We develop a model based on a seq2seq recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech prosody and the gestures have been annotated. Because the output of the model is a sequence, we use a sequence comparison technique to evaluate the model performance. We find that the model can predict certain gesture classes. In our experiment, we also replace some input features with random values to find which prosody features are pertinent. We find that the F0 is pertinent. Lastly, we also train the model on one speaker and test it with the other speaker to find whether the model is generalisable. We find that the models which we train on one speaker also works for another speaker of the same conversation.
Fichier principal
Vignette du fichier
wacai_2020_7_.pdf (489.48 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02933487 , version 1 (08-09-2020)

Identifiants

  • HAL Id : hal-02933487 , version 1

Citer

Fajrian Yunus, Chloé Clavel, Catherine I Pelachaud. Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures. Workshop sur les Affects, Compagnons artificiels et Interactions, CNRS, Université Toulouse Jean Jaurès, Université de Bordeaux, Jun 2020, Saint Pierre d'Oléron, France. ⟨hal-02933487⟩
140 Consultations
148 Téléchargements

Partager

Gmail Facebook X LinkedIn More