Georgia Institute of Technology College of Sciences

Abstract:

Low-rank adaptation (LoRA) has become the de-facto state-of-the-art method for parameter efficient fine-tuning of large-scale, pre-trained neural networks.  Similarly, low-rank compression of pre-trained networks has become a widely adopted technique to reduce the parameter count of networks for fast inference on resource constraint devices.  The idea of low-rank methods is based upon the assumption that the weight matrices of overparametrized neural networks are of low-rank.  Thus, a factorization of the weight layers based on truncated singular value decompositions can be employed to reduce the memory footprint of the network.  However, LoRA and its extensions face several challenges in practice, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process.  In this talk, Dr. Schotthoefer investigates mathematical concepts of low-rank training and uses the gained insights to design efficient and robust low-rank training algorithms.

Speaker’s Bio:

Dr. Steffen Schotthoefer is the current Householder Fellow in the Mathematics in Computation Section at the Oak Ridge National Laboratory (ORNL), affiliated with the Multiscale Methods and Dynamics Group.  Steffen's work centers on creating efficient numerical methods for training and fine-tuning artificial intelligence models in environments with limited resources and at large scales.  He investigates low-rank methods for model compression to minimize the computational cost of neural network training and inference.  In addition, Steffen develops neural network-based surrogate models for scientific domains such as radiation transport and plasma dynamics.  His research aims to tackle the challenges posed by memory and communication bottlenecks in large-scale simulations.  Prior to joining ORNL, Steffen completed his Ph.D. in Applied Mathematics at Karlsruhe Institute of Technology, Germany, focusing on neural network-based surrogate modeling for radiation transport.  During his doctoral studies, he devised numerical methods for the simulation of kinetic partial differential equations and neural network training, establishing the foundation for his current research.

Uncertainty is ubiquitous: both data and physical models inherently contain uncertainty. Therefore, it is crucial to identify the sources of uncertainty and control its propagation over time. In this talk, I will introduce two approaches to address this uncertainty propagation problem—one for the inverse problem and one for the forward problem. The main idea is to work directly with probability measures, treating the underlying PDE as a pushforward map. In the inverse setting, we will explore various variational formulations, focusing on the characterization of minimizers and their stability. In the forward setting, we aim to propose a new approach to tackle high-dimensional uncertainties.

Multimodal contrastive learning is a methodology for linking different data modalities, such as images and text. It is typically framed as the identification of a set of encoders—one for each modality—that align representations within a common latent space. In this presentation, we interpret contrastive learning as the optimization of encoders that define conditional probability distributions, for each modality conditioned on the other, in a way consistent with the available data. This probabilistic perspective suggests two natural generalizations of contrastive learning: (i) the introduction of novel probabilistic loss functions, and (ii) the use of alternative metrics for measuring alignment in the common latent space. We investigate these generalizations of the classical approach in the multivariate Gaussian setting by viewing latent space identification as a low-rank matrix approximation problem. The proposed framework is further studied through numerical experiments on multivariate Gaussians, the labeled MNIST dataset, and a data assimilation application in oceanography.

Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, natural language processing, and many other domains. The involved deep neural network architectures and computational issues have been well studied in machine learning. But there is much less theoretical understanding about the modelling, approximation or generalization abilities of deep learning models with network architectures. An important family of structured deep neural networks is deep convolutional neural networks (CNNs) induced by convolutions. The convolutional architecture gives essential differences between deep CNNs and fully-connected neural networks, and the classical approximation theory for fully-connected networks developed around 30 years ago does not apply.  This talk describes approximation and generalization analysis of deep CNNs and related structured deep neural networks.