Low-dimensional Perturb-and-MAP approach for learning Restricted Boltzmann Machines
- Jakub Tomczak
- Szymon Zaręba
- Siamak Ravanbakhsh, Dept of Computing Science, University of Alberta
- Russ Greiner, Dept of Computing Science; PI of AICML
This paper introduces a new approach to maximum likelihood learning of the parameters of a restricted Boltzmann machine (RBM). The proposed method is based on the Perturb-and-MAP (PM) paradigm that enables sampling from the Gibbs distribution. PM is a twostep process: (i) perturb the model using Gumbel perturbations, then (ii) find the maximum a posteriori (MAP) assignment of the perturbed model. We show that under certain conditions the resulting MAP configuration of the perturbed model is an unbiased sample from the original distribution. However,this approach requires an exponential number of perturbations, which is computationally intractable. Here, we apply an approximate approach based on the first order (low-dimensional) PM to calculate the gradient of the log-likelihood in binary RBM. Our approach relies on optimizing the energy function with respect to observable and hidden variables using a greedy procedure. First, for each variable we determine whether flipping this value will decrease the energy, and then we utilize the new local maximum to approximate the gradient. Moreover, we show that in some cases our approach works better than the standard coordinate-descent procedure for finding the MAP assignment and compare it with the Contrastive Divergence algorithm. We investigate the quality of our approach empirically, first on toy problems, then on various image datasets and a text dataset.
Citation
J. Tomczak, S. Zaręba, S. Ravanbakhsh, R. Greiner. "Low-dimensional Perturb-and-MAP approach for learning Restricted Boltzmann Machines". Neural Processing Letters, 50(2), pp 1401-1419, October 2018.| Keywords: | PGM, Boltzman machine, Machine Learning, Unsupervised deep learning, Gumbel perturbation, Restricted Boltzmann machine, Greedy optimization |
| Category: | In Journal |
| Web Links: | Alternative Springer |
| DOI | |
| Springer link |
BibTeX
@article{Tomczak+al:NEPL18,
author = {Jakub Tomczak and Szymon Zaręba and Siamak Ravanbakhsh and Russ
Greiner},
title = {Low-dimensional Perturb-and-MAP approach for learning Restricted
Boltzmann Machines},
Volume = "50",
Number = "2",
Pages = {1401-1419},
journal = {Neural Processing Letters},
year = 2018,
}Last Updated: February 06, 2020Submitted by Sabina P
