Seminar by Subhabrata Sen

Please Note: This seminar will be given in person.

Mean-field approximations for high-dimensional Bayesian Regression

Variational approximations provide an attractive computational alternative to MCMC-based strategies for approximating the posterior distribution in Bayesian inference. Despite their popularity in applications, supporting theoretical guarantees are limited, particularly in high-dimensional settings. In the first part of the talk, we will study bayesian inference in the context of a linear model with product priors, and derive sufficient conditions for the correctness (to leading order) of the naive mean-field approximation.

To this end, we will utilize recent advances in the theory of non-linear large deviations (Chatterjee and Dembo 2014). Next, we analyze the naive mean-field variational problem, and precisely characterize the asymptotic properties of the posterior distribution in this setting.

In the second part of the talk, we will turn to linear regression with iid gaussian design under a proportional asymptotic setting. The naive mean-field approximation is conjectured to be inaccurate in this case---instead, the Thouless-Anderson-Palmer approximation from statistical physics is expected to provide a tight approximation. We will rigorously establish the TAP formula under a uniform spherical prior on the regression coefficients.

This is based on joint work with Sumit Mukherjee (Columbia University) and Jiaze Qiu (Harvard University).