Please note: This seminar will take place in DC 1304.
Frederik Kunstner, Marie Skłodowska-Curie Postdoctoral Fellow
National Institute for Research in Digital Science and Technology (Inria)
Machine learning has had a massive impact on technology, with deep learning enabling remarkable applications in the last decade. But as the models we are trying to fit become more complex, so do the methods we use to train them. Training relies on heuristics that we, as a field, have developed through trial and error, but do not fully understand. The lack of good mental models for why those methods work makes them hard to teach and improve upon.
In this talk, I will discuss my work on the Adam algorithm, the now-default optimizer used in deep learning. I will present results from a series of papers that challenge common explanations for the success of Adam and show that its success in training language models is due to addressing a bottleneck coming from features of language data, making it possible to design more efficient methods. I will conclude by discussing my future research directions at the intersection of optimization and machine learning, focusing on building a strong understanding of why common heuristics work and how to diagnose training bottlenecks.
Bio: Frederik Kunstner is a Marie Skłodowska-Curie postdoctoral fellow at INRIA working with Francis Bach. His research combines optimization theory and empirical methods to build a better understanding on how to train ML models. He obtained his PhD from the University of British Columbia advised by Mark Schmidt, where his thesis received the CAIAC Best Doctoral Dissertation Award and a AAAI/ACM SIGAI Honorable Mention. His work on the EM algorithm won the AISTATS 2021 best paper award.