Georgia Institute of Technology College of Sciences

This dissertation focuses on neural network approximations of systems that have low dimensional structure, specifically for generative modeling and latent dynamics approximation. 

First, we establish an approximation framework for push-forward deep generative models under the manifold hypothesis. Given samples from a target probability measure supported on a low-dimensional manifold embedded in Euclidean space, we construct a neural network such that the push forward of an easy-to-sample measure by that network is close in Wasserstein metric to the target measure. The construction decomposes the target measure into local measures supported on local charts of the manifold and generates these local measures using optimal transportation theory. Crucially, the constructed network size scales with the intrinsic dimension of the manifold rather than the ambient dimension.

Next, we move on to latent dynamics learning, particularly for surrogate modeling. Given example trajectories, our goal is to construct a neural network which can autoregressively predict the evolution of a given unseen initial condition. To do so, we developed WELDNet (which stands for Windowed autoEncoders for Learning Dynamics with Neural Networks). In this approach, the time domain is segmented into overlapping regions called windows, upon which autoencoder networks are trained to compress the data to low dimensional latent space and propagator networks are trained to learn the induced time stepping map on latent space. The different windows are connected by transcoder neural networks which translate between two latent space representations of the same data. We established an approximation theory for WELDNet and performed numerical experiments on one- and two-dimensional evolutionary PDEs to show the advantage of this windowed approach compared to state-of-the-art baselines.

Zoom Link:  https://gatech.zoom.us/j/95312570686?pwd=nB4jufmtD17CBXuiRBeJS1fh4RnlHm.1 

This dissertation develops algorithms and theory for optimization, sampling, and generative modeling on manifolds, with Lie groups as a central object of study. Lie groups are manifolds with additional group structure; when endowed with a left-invariant Riemannian metric they become Riemannian manifolds, a setting that plays a central role throughout this work.

We first consider optimization on the Stiefel manifold $\mathrm{St}(n,d)$, the set of $n\times d$ matrices with orthonormal columns. By deriving a variational principle on this manifold, we construct the Momentum Stiefel Optimizer, a momentum-based algorithm that exactly preserves the orthogonality constraint at every iteration. The optimizer is applied to suitably-orthogonal attention in transformers and to optimal transport problems, achieving consistent improvements over existing methods.

We then establish quantitative convergence guarantees for momentum optimizers on Lie groups equipped with a left-invariant metric. Using the left-trivialization technique, which maps the curved dynamics to a flat Euclidean space for the momentum variable, we prove the first explicit convergence rates for both Heavy-Ball and Nesterov Accelerated methods on compact Lie groups, with rates that match Euclidean theory in terms of the smoothness and strong-convexity constants.

Next, we develop gauge-equivariant accelerated methods for optimization over the Grassmannian $\mathrm{Gr}(n,d)$, the set of $d$-dimensional subspaces of $\mathbb{R}^n$. Because each subspace has infinitely many orthonormal representatives related by an $\mathrm{O}(d)$ rotation, a naive lift of Stiefel algorithms to the Grassmannian is not gauge-equivariant. We introduce a gauge-fixing pipeline that converts any Stiefel optimizer into a gauge-equivariant Grassmann algorithm, yielding Grassmann Anderson Acceleration and Grassmann NAG, validated on density functional theory and low-rank matrix completion.

We then turn to sampling on Lie groups. By adding tractable noise to the left-trivialized momentum dynamics, we construct the first kinetic (momentum) Langevin Monte Carlo sampler on Lie groups with rigorous nonasymptotic convergence guarantees. The sampler preserves the group structure exactly at every step. Exponential convergence in $W_2$ distance is proved under only compactness of the Lie group and geodesic smoothness of the potential, without any convexity or isoperimetric assumption.

Finally, we address score-based generative modeling on general Riemannian manifolds. The standard denoising score matching framework requires a tractable forward-process transition kernel, which is unavailable on general manifolds because the heat kernel is intractable. We propose splitting diffusion, which lifts the dynamics to the tangent bundle and alternates closed-form stochastic momentum updates in the Euclidean tangent space with deterministic geodesic transport. The resulting transition kernel is closed-form, enabling denoising score matching training on general Riemannian manifolds requiring only exponential map as oracle.