Georgia Institute of Technology College of Sciences

Abstract:

Low-rank adaptation (LoRA) has become the de-facto state-of-the-art method for parameter efficient fine-tuning of large-scale, pre-trained neural networks.  Similarly, low-rank compression of pre-trained networks has become a widely adopted technique to reduce the parameter count of networks for fast inference on resource constraint devices.  The idea of low-rank methods is based upon the assumption that the weight matrices of overparametrized neural networks are of low-rank.  Thus, a factorization of the weight layers based on truncated singular value decompositions can be employed to reduce the memory footprint of the network.  However, LoRA and its extensions face several challenges in practice, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process.  In this talk, Dr. Schotthoefer investigates mathematical concepts of low-rank training and uses the gained insights to design efficient and robust low-rank training algorithms.

Speaker’s Bio:

Dr. Steffen Schotthoefer is the current Householder Fellow in the Mathematics in Computation Section at the Oak Ridge National Laboratory (ORNL), affiliated with the Multiscale Methods and Dynamics Group.  Steffen's work centers on creating efficient numerical methods for training and fine-tuning artificial intelligence models in environments with limited resources and at large scales.  He investigates low-rank methods for model compression to minimize the computational cost of neural network training and inference.  In addition, Steffen develops neural network-based surrogate models for scientific domains such as radiation transport and plasma dynamics.  His research aims to tackle the challenges posed by memory and communication bottlenecks in large-scale simulations.  Prior to joining ORNL, Steffen completed his Ph.D. in Applied Mathematics at Karlsruhe Institute of Technology, Germany, focusing on neural network-based surrogate modeling for radiation transport.  During his doctoral studies, he devised numerical methods for the simulation of kinetic partial differential equations and neural network training, establishing the foundation for his current research.

We will start with a 15-minute presentation by Noah Solomon and continue with a free discussion.

Dehn surgery (with a fixed slope p/q) associates, to a knot in S^3, a 3-manifold M with first homology isomorphic to the integers mod p. One might wonder if this function is one-to-one or onto; Cameron Gordon (1978) conjectured that it is never injective nor surjective. The surjectivity case was established a decade later, while the injectivity case was only recently proven by Hayden, Piccirillo, and Wakelin. We will survey this latter result and its proof. 

I will discuss several classical problems in number theory about representation of numbers and forms by quadratic forms and related counting results. We will describe several applications such as pair correlations and eigenvalue statistics for quantum systems.
Then we will move to related problems about irrational forms, such as the Oppenheim conjecture and explain how homogeneous dynamics can help to tackle such problems.
The talk will be self contained and hopefully accessible.