Speaker: Roger Grosse, University of Toronto
Title: Exploiting compositional structure for Bayesian inference, model selection, and optimization
I will present flexible algorithms for model discovery and model fitting which apply to broad, open-ended classes of models, yet which also incorporate model-specific algorithmic insights. First, I will introduce a framework for building probabilistic models compositionally out of common modeling motifs, such as clustering, sparsity, and dimensionality reduction. This compositional framework yields a variety of existing models as special cases. We can flexibly perform posterior inference across this large, open-ended space of models by composing sophisticated inference algorithms carefully designed for the individual modeling motifs. An automatic structure search procedure over this space of models yields sensible analyses of datasets as diverse as motion capture, natural image patches, and Senate voting records, all using a single software package with no hand-tuned metaparameters.
In the second part of my talk, I will focus on deriving efficient optimizers from the structure of a model's computation graph. Natural gradient descent has the potential to greatly speed up the training of neural networks by correcting for the curvature of the loss function, but the exact method is impractical because it requires solving a (possibly million-dimensional) dense linear system involving the Fisher matrix. I will present a technique for deriving scalable yet highly expressive approximations to the Fisher matrix by fitting structured probabilistic models to the computation of the log-likelihood derivatives. Because of the method's invariance properties, it automatically gains the computational advantages of recently proposed reparameterizations of neural networks. It greatly speeds up the training of restricted Boltzmann machines, deep autoencoders, and convolutional networks.
Roger Grosse is a Postdoctoral Fellow in the University of Toronto machine learning group. He received his Ph.D. in computer science from MIT under the supervision of of Bill Freeman. He is a recipient of the NDSEG Graduate Fellowship, the Banting Postdoctoral Fellowship, and outstanding paper awards at the International Conference of Machine Learning (ICML) and the Conference for Uncertainty in AI (UAI). He is also a co-creator of Metacademy, an open-source web site for developing personalized learning plans in machine learning and related fields.