Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Infinite-dimensional gradient-based descent for alpha-divergence minimisation

Kamélia Daudel 1, 2 Randal Douc 3, 4 François Portier 1, 2
2 S2A - Signal, Statistique et Apprentissage
LTCI - Laboratoire Traitement et Communication de l'Information
4 TIPIC-SAMOVAR - Traitement de l'Information Pour Images et Communications
SAMOVAR - Services répartis, Architectures, MOdélisation, Validation, Administration des Réseaux
Abstract : This paper introduces the $(\alpha, \Gamma)$-descent, an iterative algorithm which operates on measures and performs $\alpha$-divergence minimisation in a Bayesian framework. This gradient-based procedure extends the commonly-used variational approximation by adding a prior on the variational parameters in the form of a measure. We prove that for a rich family of functions $\Gamma$, this algorithm leads at each step to a systematic decrease in the $\alpha$-divergence and derive convergence results. Our framework recovers the Entropic Mirror Descent algorithm and provides an alternative algorithm that we call the Power Descent. Moreover, in its stochastic formulation, the $(\alpha, \Gamma)$-descent allows to optimise the mixture weights of any given mixture model without any information on the underlying distribution of the variational parameters. This renders our method compatible with many choices of parameters updates and applicable to a wide range of Machine Learning tasks. We demonstrate empirically on both toy and real-world examples the benefit of using the Power descent and going beyond the Entropic Mirror Descent framework, which fails as the dimension grows.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [54 references]  Display  Hide  Download

https://hal.telecom-paris.fr/hal-02614605
Contributor : Kamélia Daudel <>
Submitted on : Thursday, October 15, 2020 - 9:30:29 AM
Last modification on : Monday, October 19, 2020 - 3:15:42 AM

File

DaudelDoucPortier2020.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02614605, version 2
  • ARXIV : 2005.10618

Citation

Kamélia Daudel, Randal Douc, François Portier. Infinite-dimensional gradient-based descent for alpha-divergence minimisation. 2020. ⟨hal-02614605v2⟩

Share

Metrics

Record views

52

Files downloads

19