Georgia Institute of Technology College of Sciences

Transformers serve as the foundational architecture for large language and video generation models, such as GPT, BERT, SORA, and their successors. While empirical studies have shown that real-world data and learning tasks exhibit low-dimensional geometric structures, the theoretical understanding of transformers in leveraging these structures remains largely unexplored. In this talk, we present a theoretical foundation for transformers in two key scenarios: (1) regression tasks with noisy input data lying near a low-dimensional manifold, and (2) in-context learning (ICL) for regression of Hölder functions on manifolds. For the first setting, we prove that approximation and generalization bound that depend crucially on the intrinsic dimension of the manifold, demonstrating that transformers can effectively learn from data perturbed by high-dimensional noise. For the second setting, we derive generalization error bounds for ICL in terms of prompt length and the number of training tasks, revealing that transformers achieve the minimax optimal rate for Hölder regression—scaling exponentially with the intrinsic rather than ambient dimension. Together, these results provide foundational insights into how transformers exploit low-dimensional geometric structures in learning tasks, advancing our theoretical understanding of their remarkable empirical success.

Classical solvers for large-scale scientific and data-driven problems often face limitations when uncertainty, multiscale effects, or ill-conditioning become dominant. In this talk, I will present hybrid algorithmic frameworks that unify ideas from numerical analysis, stochastic computation, and machine learning to address these challenges. In the first part, I will introduce Preconditioned Truncated Single-Sample (PTSS) estimators, a new class of stochastic Krylov methods that integrate preconditioning with truncated Lanczos iterations. PTSS provides unbiased, low-variance estimators for linear system solutions, log-determinants, and their derivatives, enabling scalable algorithms for inference and optimization. In the second part, I will discuss a data-driven approach to constructing approximate inverse preconditioners for partial differential equations (PDEs). By learning the Green’s function of the underlying operator through neural representations, this framework captures multiscale behavior and preserves essential spectral structure. The resulting solvers achieve near-linear complexity in both setup and application. Together, these developments illustrate how stochastic and learning-based mechanisms can be embedded into classical numerical frameworks to create adaptive and efficient computational methods for complex systems.

Abstract:

Low-rank adaptation (LoRA) has become the de-facto state-of-the-art method for parameter efficient fine-tuning of large-scale, pre-trained neural networks.  Similarly, low-rank compression of pre-trained networks has become a widely adopted technique to reduce the parameter count of networks for fast inference on resource constraint devices.  The idea of low-rank methods is based upon the assumption that the weight matrices of overparametrized neural networks are of low-rank.  Thus, a factorization of the weight layers based on truncated singular value decompositions can be employed to reduce the memory footprint of the network.  However, LoRA and its extensions face several challenges in practice, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process.  In this talk, Dr. Schotthoefer investigates mathematical concepts of low-rank training and uses the gained insights to design efficient and robust low-rank training algorithms.

Speaker’s Bio:

Dr. Steffen Schotthoefer is the current Householder Fellow in the Mathematics in Computation Section at the Oak Ridge National Laboratory (ORNL), affiliated with the Multiscale Methods and Dynamics Group.  Steffen's work centers on creating efficient numerical methods for training and fine-tuning artificial intelligence models in environments with limited resources and at large scales.  He investigates low-rank methods for model compression to minimize the computational cost of neural network training and inference.  In addition, Steffen develops neural network-based surrogate models for scientific domains such as radiation transport and plasma dynamics.  His research aims to tackle the challenges posed by memory and communication bottlenecks in large-scale simulations.  Prior to joining ORNL, Steffen completed his Ph.D. in Applied Mathematics at Karlsruhe Institute of Technology, Germany, focusing on neural network-based surrogate modeling for radiation transport.  During his doctoral studies, he devised numerical methods for the simulation of kinetic partial differential equations and neural network training, establishing the foundation for his current research.

Uncertainty is ubiquitous: both data and physical models inherently contain uncertainty. Therefore, it is crucial to identify the sources of uncertainty and control its propagation over time. In this talk, I will introduce two approaches to address this uncertainty propagation problem—one for the inverse problem and one for the forward problem. The main idea is to work directly with probability measures, treating the underlying PDE as a pushforward map. In the inverse setting, we will explore various variational formulations, focusing on the characterization of minimizers and their stability. In the forward setting, we aim to propose a new approach to tackle high-dimensional uncertainties.