Georgia Institute of Technology College of Sciences

Convex relaxations are of central interest in optimization, and it is typically challenging to determine whether a given convex relaxation will be tight for a given problem. We introduce a topological framework for analyzing  situations in which a constrained optimization problem over a nonconvex set (such as a manifold) has a tight convex relaxation. In particular, we give a criterion for the existence of such a tight convex relaxation in terms of the existence of a continuous function of Lagrange multipliers for the constrained problem maximizing the corresponding Lagrangian. We call such a function a Lagrangian dual section, in reference to the topological notion of a section of a bundle.

As a corollary of this result, we will give new criteria for the exactness of SDP relaxations for Stiefel manifold optimization and inverse eigenvalue problems in terms of linear subspaces of matrices satisfying spectral properties such as being nonsingular. We will also illustrate a homotopy continuation style algorithm with global optimality guarantees with applications to the unbalanced procrustes problem.

In a recent joint work with J. Buzzi, Y. Shi, and J. Yang, given a diffeomorphism preserving a one-dimensional expanding foliation $\mathcal F$ with homogeneous exponential growth, we construct a family of reference measures on each leaf of the foliation with controlled Jacobian and a Gibbs property.

We then prove that for any measure of maximal $\mathcal F$-entropy, its conditional measures on each leaf must be equivalent to the reference measures.

When the reference measures are equivalent to the leafwise Lebesgue measure, we prove that the log-determinant of $f$ must be cohomologous to a constant.

We will consider several applications, including the strong and center foliations of Anosov diffeomorphisms, factor over Anosov diffeomorphisms, and perturbations of the time-one map of geodesic flows on surfaces with negative curvature. We will also discuss several conjectures on the unique ergodicity and (exponential) equidistribution for the strong unstable foliations of Anosov systems. 

Zoom link: https://gatech.zoom.us/j/92005780980?pwd=ptlx7KdBAbHI3DTvv6V7fjFn27LDaE.1

Meeting ID: 920 0578 0980
Passcode: 604975

Abstract:

Low-rank adaptation (LoRA) has become the de-facto state-of-the-art method for parameter efficient fine-tuning of large-scale, pre-trained neural networks.  Similarly, low-rank compression of pre-trained networks has become a widely adopted technique to reduce the parameter count of networks for fast inference on resource constraint devices.  The idea of low-rank methods is based upon the assumption that the weight matrices of overparametrized neural networks are of low-rank.  Thus, a factorization of the weight layers based on truncated singular value decompositions can be employed to reduce the memory footprint of the network.  However, LoRA and its extensions face several challenges in practice, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process.  In this talk, Dr. Schotthoefer investigates mathematical concepts of low-rank training and uses the gained insights to design efficient and robust low-rank training algorithms.

Speaker’s Bio:

Dr. Steffen Schotthoefer is the current Householder Fellow in the Mathematics in Computation Section at the Oak Ridge National Laboratory (ORNL), affiliated with the Multiscale Methods and Dynamics Group.  Steffen's work centers on creating efficient numerical methods for training and fine-tuning artificial intelligence models in environments with limited resources and at large scales.  He investigates low-rank methods for model compression to minimize the computational cost of neural network training and inference.  In addition, Steffen develops neural network-based surrogate models for scientific domains such as radiation transport and plasma dynamics.  His research aims to tackle the challenges posed by memory and communication bottlenecks in large-scale simulations.  Prior to joining ORNL, Steffen completed his Ph.D. in Applied Mathematics at Karlsruhe Institute of Technology, Germany, focusing on neural network-based surrogate modeling for radiation transport.  During his doctoral studies, he devised numerical methods for the simulation of kinetic partial differential equations and neural network training, establishing the foundation for his current research.

We will start with a 15-minute presentation by Noah Solomon and continue with a free discussion.