Georgia Institute of Technology College of Sciences

In this talk, I will discuss our recent theoretical advancements in generative modeling. The first part of the presentation will focus on learning distributions with symmetry. I will introduce results on the sample complexity of empirical estimations of probability divergences for group-invariant distributions, and present performance guarantees for GANs and score-based generative models that incorporate symmetry. Notably, I will offer the first quantitative comparison between data augmentation and directly embedding symmetry into models, highlighting the latter as a more fundamental approach for efficient learning. These findings underscore how incorporating symmetry into generative models can significantly enhance learning efficiency, particularly in data-limited scenarios. The second part will cover $\alpha$-divergences with Wasserstein-1 regularization. These divergences can be interpreted as $\alpha$-divergences constrained to Lipschitz test functions in their variational form. I will demonstrate how generative learning can be made agnostic to assumptions about target distributions, including those with heavy tails or low-dimensional and fractal supports, through the use of these divergences as objective functionals. I will outline the conditions for the finiteness of these divergences under minimal assumptions on the target distribution along with the variational derivatives and gradient flow formulation associated with them. This framework provides guarantees for various machine learning algorithms that optimize over this class of divergences. 

In this talk, I will present a damped proximal augmented Lagrangian method (DPALM) for solving problems with a weakly-convex objective and convex linear/nonlinear constraints. Instead of taking a full stepsize, DPALM adopts a damped dual stepsize. DPALM can produce a (near) eps-KKT point within eps^{-2} outer iterations if each DPALM subproblem is solved to a proper accuracy. In addition, I will show overall iteration complexity of DPALM when the objective is either a regularized smooth function or in a regularized compositional form. For the former case, DPALM achieves the complexity of eps^{-2.5} to produce an eps-KKT point by applying an accelerated proximal gradient (APG) method to each DPALM subproblem. For the latter case, the complexity of DPALM is eps^{-3} to produce a near eps-KKT point by using an APG to solve a Moreau-envelope smoothed version of each subproblem. Our outer iteration complexity and the overall complexity either generalize existing best ones from unconstrained or linear-constrained problems to convex-constrained ones, or improve over the best-known results on solving the same-structured problems. Furthermore, numerical experiments on linearly/quadratically constrained non-convex quadratic programs and linear-constrained robust nonlinear least squares are conducted to demonstrate the empirical efficiency of the proposed DPALM over several state-of-the art methods.

Spatio-temporal modeling of real-world data presents significant challenges due to high-dimensionality, noisy measurements, and limited data. In this talk, we introduce two frameworks that jointly solve the problems of sparse identification of governing equations and latent space reconstruction: the Bayesian SINDy autoencoder and SINDy-SHRED. The Bayesian SINDy autoencoder leverages a spike-and-slab prior to enable robust discovery of governing equations and latent coordinate systems, providing uncertainty estimates in low-data, high-noise settings. In our experiments, we applied the Bayesian SINDy autoencoder to real video data, marking the first example of learning governing equations directly from such data. This framework successfully identified underlying physical laws, such as accurately estimating constants like gravity from pendulum videos, even in the presence of noise and limited samples.

In parallel, SINDy-SHRED integrates Gated Recurrent Units (GRUs) with a shallow decoder network to model temporal sequences and reconstruct full spatio-temporal fields using only a few sensors. Our proposed algorithm introduces a SINDy-based regularization. Beginning with an arbitrary latent state space, the dynamics of the latent space progressively converges to a SINDy-class functional. We conduct a systematic experimental study including synthetic PDE data, real-world sensor measurements for sea surface temperature, and direct video data. With no explicit encoder, SINDy-SHRED allows for efficient training with minimal hyperparameter tuning and laptop-level computing. SINDy-SHRED demonstrates robust generalization in a variety of applications with minimal to no hyperparameter adjustments. Additionally, the interpretable SINDy model of latent state dynamics enables accurate long-term video predictions, achieving state-of-the-art performance and outperforming all baseline methods considered, including Convolutional LSTM, PredRNN, ResNet, and SimVP.