Georgia Institute of Technology College of Sciences

Abstract: Critical points significantly affect the behavior of gradient-based dynamics. Numerous works have been done for global minima of neural networks. Thus, the recent work characterizes non-global critical points. With the idea that gradient-based methods of neural networks favor “simple models”, this work focuses on the set of low-complexity critical points, i.e., those representing underparameterized network models. Specifically, we investigate: i) the existence and ii) geometry of such sets, iii) the output functions they represent, iv) saddles in them. The talk will discuss these results based on a simple example. The general theorems will also be included. No specific knowledge in neural networks is required. 

Let $G = (V,E)$ be a graph on $n$ vertices, and let $c : E \to P$, where $P$ is a set of colors. Let $\delta^c(G) = \min_{v \in V} \{ d^{c}(v) \}$ where $d^c(v)$ is the number of colors on edges incident to a vertex $v$ of $G$.  In 2011, Fujita and Magnant showed that if $G$ is a graph on $n$ vertices that satisfies $\delta^c(G)\geq n/2$, then for every two vertices $u, v$ there is a properly-colored $u,v$-path in $G$. We show that for sufficiently large graphs $G$ the same bound for $\delta^c(G)$ implies that any two vertices are connected by a rainbow path. This is joint work with Andrzej Czygrinow.

Machine learning (ML) has achieved unprecedented empirical success in diverse applications. It now has been applied to solve scientific and engineering problems, which has become an emerging field, Scientific Machine Learning (SciML). However, many ML techniques are highly complex and sophisticated, often requiring extensive trial-and-error experimentation and specialized problem-dependent tricks to implement effectively. This complexity frequently leads to significant challenges, such as reproducibility and rigorness, for scientific research. This talk explores mathematical approaches, offering more principled and reliable methodologies in SciML. The first part will present recent efforts advancing the predictive power of physics-informed machine learning through robust training/optimization methods. This includes an effective training method for multivariate neural networks, namely, Active Neuron Least Squares (ANLS) and a two-step training method for deep operator networks. The second part is about how to embed the first principles of physics into neural networks. I will present a general framework for designing NNs that obey the first and second laws of thermodynamics. The framework not only provides flexible ways of leveraging available physics information but also results in expressive NN architectures. I will also present an intriguing phenomenon of this framework when it is applied in the context of latent space dynamics identification where a correlation appears between an entropy production rate in the latent space and the behaviors of the full-state solution.