Location
MC 5501
Speaker
Aukosh Jagannath, University of Waterloo
Title
The training dynamics and local geometry of high-dimensional learning
Abstract
Many modern data science tasks involve optimizing a complex, random function in high dimensions. The go-to methods for such problems are variations of stochastic gradient descent (SGD). Though they perform remarkably well, the rigorous analysis of SGD on high-dimensional statistical models is in its infancy. This talk will study the high-dimensional limits of the training dynamics and the local geometry from the point-of-view of the algorithm. We will discuss how in this limit, for many statistical tasks, the evolution of a certain summary statistics converge to a closed finite dimensional dynamical system, called their effective dynamics. If there is time, we will then discuss some applications connecting this to an understanding of the spectrum of the Hessian and Information matrices along the training trajectory. This talk is based on a series of joint works with G. Ben Arous (NYU), R. Gheissari (Northwestern), J. Huang (U Penn), T. Jones-McCormick (Waterloo), and V. Sarangian (Waterloo).