Category: Machine Learning and Statistics

Machine Learning and Statistics5 Comments

Variational Inference: Tricks of the Trade

shakirm

The NIPS 2014 Workshop on Advances in Variational Inference was abuzz with new methods and ideas for scalable approximate inference. The concluding event of the workshop was a lively debate with David Blei, Neil Lawrence, Zoubin Ghahramani, Shinichi Nakajima and Matthias Seeger on the history, trends and open questions in variational inference. One of the questions posed to our panel and audience was: 'what are your variational inference tricks-of-the-trade?'

My current best-practice at present includes: stochastic approximation, Monte Carlo estimation, amortised inference and powerful software tools. But this is a though-provoking question that has has motivated me think in some more detail through my current variational inference tricks-of-the-trade, which are:
Continue reading "Variational Inference: Tricks of the Trade"

Machine Learning and StatisticsLeave a comment

Bayesian sparsity using spike-and-slab priors

shakirm

I recently received some queries on our paper: S. Mohamed, K. Heller and Z. Ghahramani. Bayesian and L1 Approaches for Sparse Unsupervised Learning. International Conference on Machine Learning (ICML), June 2012 [cite key="mohamed2012sparse"]. The questions were very good and I thought it would be useful to post these for future reference. The paper looked at Bayesian and optimisation approaches for learning sparse models. For Bayesian models, we advocated the use of spike-and-slab sparse models and specified an adapted latent Gaussian model with an additional set of discrete latent variables to specify when a latent dimension is sparse or not. This … Continue reading Bayesian sparsity using spike-and-slab priors

Machine Learning and Statistics6 Comments

Marr's Levels of Analysis

shakirm
Marr's 3 levels of analysis.
Marr's three levels of analysis.

Marr's three levels of analysis [cite key="marr1982"] promotes the idea that complex systems such as the brain, a computer or human behaviour should be understood at different levels. Marr's framework proved to be an elegant and popular way of reasoning about complex systems, and in the context of machine learning and statistics,  remains an intuitive framework that is often used when describing probabilistic models of cognition and perceptual systems.

Continue reading "Marr's Levels of Analysis"

Machine Learning and Statistics3 Comments

On Marginal Likelihoods and Widely Applicable BIC

shakirm

The marginal likelihood of a model is one the key quantities appearing throughout machine learning and statistics, since it provides an intuitive and natural objective function for model selection and parameter estimation. I recently read a new paper by Sumio Watanabe on A Widely applicable Bayesian information criterion (WBIC)[cite key="watanabe2012widely"] (and to appear in JMLR soon) that  provides a new, theoretically grounded and easy to implement method of approximating the marginal likelihood, which I will briefly describe in this post. I'll summarise some of the important aspects of the marginal likelihood and then briefly describe the WBIC and some thoughts and questions on its use.

Continue reading "On Marginal Likelihoods and Widely Applicable BIC"