Georgia Institute of Technology College of Sciences

Most modern machine learning applications are based on overparameterized neural networks trained by variants of stochastic gradient descent. To explain the performance of these networks from a theoretical perspective (in particular the so-called "implicit bias"), it is necessary to understand the random dynamics of the optimization algorithms. Mathematically this amounts to the study of random dynamical systems with manifolds of equilibria. In this talk, I will give a brief introduction to machine learning theory and explain how almost-sure Lyapunov exponents and moment Lyapunov exponents can be used to characterize the set of possible limit points for stochastic gradient descent.