Skip to Main content Skip to Navigation
Conference papers

Click to Move: Controlling Video Generation with Sparse Motion

Abstract : This paper introduces Click to Move (C2M), a novel framework for video generation where the user can control the motion of the synthesized video through mouse clicks specifying simple object trajectories of the key objects in the scene. Our model receives as input an initial frame, its corresponding segmentation map and the sparse motion vectors encoding the input provided by the user. It outputs a plausible video sequence starting from the given frame and with a motion that is consistent with user input. Notably, our proposed deep architecture incorporates a Graph Convolution Network (GCN) modelling the movements of all the objects in the scene in a holistic manner and effectively combining the sparse user motion information and image features. Experimental results show that C2M outperforms existing methods on two publicly available datasets, thus demonstrating the effectiveness of our GCN framework at modelling object interactions. The source code is publicly available at https://github.com/PierfrancescoArdino/C2M.
Document type :
Conference papers
Complete list of metadata

https://hal.telecom-paris.fr/hal-03335971
Contributor : Stéphane Lathuilière Connect in order to contact the contributor
Submitted on : Monday, September 6, 2021 - 4:29:16 PM
Last modification on : Tuesday, October 19, 2021 - 11:15:25 AM

Links full text

Identifiers

  • HAL Id : hal-03335971, version 1
  • ARXIV : 2108.08815

Collections

Citation

Pierfrancesco Ardino, Marco de Nadai, Bruno Lepri, Elisa Ricci, Stéphane Lathuilière. Click to Move: Controlling Video Generation with Sparse Motion. International Conference on Computer Vision, Oct 2021, Online, France. ⟨hal-03335971⟩

Share

Metrics

Record views

15