Georgia Institute of Technology College of Sciences

Abstract: Critical points significantly affect the behavior of gradient-based dynamics. Numerous works have been done for global minima of neural networks. Thus, the recent work characterizes non-global critical points. With the idea that gradient-based methods of neural networks favor “simple models”, this work focuses on the set of low-complexity critical points, i.e., those representing underparameterized network models. Specifically, we investigate: i) the existence and ii) geometry of such sets, iii) the output functions they represent, iv) saddles in them. The talk will discuss these results based on a simple example. The general theorems will also be included. No specific knowledge in neural networks is required.