Georgia Institute of Technology College of Sciences

Leveraging large-scale data and systems of computing accelerators, statistical learning has led to  significant paradigm shifts in many scientific disciplines. Grand challenges in science have been tackled with exciting synergy between disciplinary science, physics-based simulations via high-performance computing, and powerful learning methods.

In this talk, I will describe several vignettes of our research in the theme of modeling complex dynamical systems characterized by partial differential equations with turbulent solutions. I will also demonstrate how machine learning technologies, especially advances in generative AI technology,  are effectively applied to address the computational and modeling challenges in such systems, exemplified by their successful applications to  weather forecast and climate projection. I will also discuss what new challenges and opportunities have been brought into future machine learning research.

The research work presented in this talk is based on joint and interdisciplinary research work of several teams at Google Research, ETH and Caltech.


Bio: Dr. Fei Sha is currently a research scientist at Google Research, where he leads a team of scientists and engineers working on scientific machine learning with a specific application focus towards AI for Weather and Climate. He was a full professor and the Zohrab A. Kaprielian Fellow in Engineering at the Department of Computer Science, University of Southern California. His primary research interests are machine learning and its application to various AI problems: speech and language processing, computer vision, robotics and recently scientific computing, dynamical systems, weather forecast and climate modeling.  Dr. Sha was selected as a Alfred P. Sloan Research Fellow in 2013, and also won an Army Research Office Young Investigator Award in 2012. He has a Ph.D from Computer and Information Science from U. of Pennsylvania and B.Sc and M.Sc from Southeast University (Nanjing, China). More information about Dr. Sha's scholastic activities can be found at his microsite at http://feisha.org.

In this talk I will discuss our recent effort to develop structure-preserving machine learning (ML) for time series data, focusing on both dissipative PDEs and singularly perturbed ODEs. The first part presents a data-driven modeling method that accurately captures shocks and chaotic dynamics through a stabilized neural ODE framework. We learn the right-hand-side of an ODE by adding the outputs of two networks together, one learning a linear term and the other a nonlinear term. The architecture is inspired by the inertial manifold theorem. We apply this method to chaotic trajectories of the Kuramoto-Sivashinsky equation, where our model keeps long-term trajectories on the attractor and remains robust to noisy initial conditions. The second part explores structure-preserving ML for singularly perturbed dynamical systems. A powerful tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds. I will discuss a novel realization of this concept using ML. Specifically, a fast-slow neural network (FSNN) is proposed, enforcing the existence of a trainable, attractive invariant slow manifold as a hard constraint. To illustrate the power of FSNN, I will show a fusion-motivated example where traditional numerical integrators all fail.

In this talk, I will discuss our recent theoretical advancements in generative modeling. The first part of the presentation will focus on learning distributions with symmetry. I will introduce results on the sample complexity of empirical estimations of probability divergences for group-invariant distributions, and present performance guarantees for GANs and score-based generative models that incorporate symmetry. Notably, I will offer the first quantitative comparison between data augmentation and directly embedding symmetry into models, highlighting the latter as a more fundamental approach for efficient learning. These findings underscore how incorporating symmetry into generative models can significantly enhance learning efficiency, particularly in data-limited scenarios. The second part will cover $\alpha$-divergences with Wasserstein-1 regularization. These divergences can be interpreted as $\alpha$-divergences constrained to Lipschitz test functions in their variational form. I will demonstrate how generative learning can be made agnostic to assumptions about target distributions, including those with heavy tails or low-dimensional and fractal supports, through the use of these divergences as objective functionals. I will outline the conditions for the finiteness of these divergences under minimal assumptions on the target distribution along with the variational derivatives and gradient flow formulation associated with them. This framework provides guarantees for various machine learning algorithms that optimize over this class of divergences. 

In this talk, I will present a damped proximal augmented Lagrangian method (DPALM) for solving problems with a weakly-convex objective and convex linear/nonlinear constraints. Instead of taking a full stepsize, DPALM adopts a damped dual stepsize. DPALM can produce a (near) eps-KKT point within eps^{-2} outer iterations if each DPALM subproblem is solved to a proper accuracy. In addition, I will show overall iteration complexity of DPALM when the objective is either a regularized smooth function or in a regularized compositional form. For the former case, DPALM achieves the complexity of eps^{-2.5} to produce an eps-KKT point by applying an accelerated proximal gradient (APG) method to each DPALM subproblem. For the latter case, the complexity of DPALM is eps^{-3} to produce a near eps-KKT point by using an APG to solve a Moreau-envelope smoothed version of each subproblem. Our outer iteration complexity and the overall complexity either generalize existing best ones from unconstrained or linear-constrained problems to convex-constrained ones, or improve over the best-known results on solving the same-structured problems. Furthermore, numerical experiments on linearly/quadratically constrained non-convex quadratic programs and linear-constrained robust nonlinear least squares are conducted to demonstrate the empirical efficiency of the proposed DPALM over several state-of-the art methods.