Dynamic Stability in Stochastic Gradient Descent

CDSNS Colloquium
Friday, May 24, 2024 - 3:30pm for
Skiles 254
Dennis Chemnitz – FU Berlin – dennis@zedat.fu-berlin.dehttps://www.mi.fu-berlin.de/en/math/groups/ag-random-dynamics/people/scientific-Staff/dennis_chemnitz.html
Alex Blumenthal

Please Note: Streaming via Zoom: https://gatech.zoom.us/j/91390791493?pwd=QnpaWHNEOHZTVXlZSXFkYTJ0b0Q0UT09

Most modern machine learning applications are based on overparameterized neural networks trained by variants of stochastic gradient descent. To explain the performance of these networks from a theoretical perspective (in particular the so-called "implicit bias"), it is necessary to understand the random dynamics of the optimization algorithms. Mathematically this amounts to the study of random dynamical systems with manifolds of equilibria. In this talk, I will give a brief introduction to machine learning theory and explain how almost-sure Lyapunov exponents and moment Lyapunov exponents can be used to characterize the set of possible limit points for stochastic gradient descent.