Seminars and Colloquia by Series

Two Phases of Scaling Laws for Nearest Neighbor Classifiers

Series
Applied and Computational Mathematics Seminar
Time
Thursday, May 25, 2023 - 10:30 for 1 hour (actually 50 minutes)
Location
https://gatech.zoom.us/j/98355006347
Speaker
Jingzhao ZhangTsinghua University

Please Note: Special time & day. Remote only.

A scaling law refers to the observation that the test performance of a model improves as the number of training data increases. A fast scaling law implies that one can solve machine learning problems by simply boosting the data and the model sizes. Yet, in many cases, the benefit of adding more data can be negligible. In this work, we study the rate of scaling laws of nearest neighbor classifiers. We show that a scaling law can have two phases: in the first phase, the generalization error depends polynomially on the data dimension and decreases fast; whereas in the second phase, the error depends exponentially on the data dimension and decreases slowly. Our analysis highlights the complexity of the data distribution in determining the generalization error. When the data distributes benignly, our result suggests that nearest neighbor classifier can achieve a generalization error that depends polynomially, instead of exponentially, on the data dimension.

Uncovering the Law of Data Separation in Deep Learning

Series
Applied and Computational Mathematics Seminar
Time
Monday, April 17, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Prof. Weijie SuUniversity of Pennsylvania (Wharton)

Please Note: The speaker will present in person.

In this talk, we will investigate the emergence of geometric patterns in well-trained deep learning models by making use of a layer-peeled model and the law of equi-separation. The former is a nonconvex optimization program that models the last-layer features and weights. We use the model to shed light on the neural collapse phenomenon of Papyan, Han, and Donoho, and to predict a hitherto-unknown phenomenon that we term minority collapse in imbalanced training.
 
The law of equi-separation is a pervasive empirical phenomenon that describes how data are separated according to their class membership from the bottom to the top layer in a well-trained neural network. We will show that, through extensive computational experiments, neural networks improve data separation through layers in a simple exponential manner. This law leads to roughly equal ratios of separation that a single layer is able to improve, thereby showing that all layers are created equal. We will conclude the talk by discussing the implications of this law on the interpretation, robustness, and generalization of deep learning, as well as on the inadequacy of some existing approaches toward demystifying deep learning.
 

Mathematical Foundations of Graph-Based Bayesian Semi-Supervised Learning

Series
Applied and Computational Mathematics Seminar
Time
Monday, April 10, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Prof. Daniel Sanz-AlonsoU Chicago

Please Note: Speaker will present in person

Semi-supervised learning refers to the problem of recovering an input-output map using many unlabeled examples and a few labeled ones. In this talk I will survey several mathematical questions arising from the Bayesian formulation of graph-based semi-supervised learning. These questions include the modeling of prior distributions for functions on graphs, the derivation of continuum limits for the posterior, the design of scalable posterior sampling algorithms, and the contraction of the posterior in the large data limit.

New gradient sliding results on convex optimization with smoothness structure

Series
Applied and Computational Mathematics Seminar
Time
Monday, April 3, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Yuyuan OuyangClemson University

In this talk, we present new gradient sliding results for constrained convex optimization with applications in image reconstruction and decentralized distributed optimization. Specifically, we will study classes of large-scale problems that minimizes a convex objective function over feasible set with linear constraints. We will show that by exploring the gradient sliding technique, the number of gradient evaluations of the objective function can be reduced by exploring the smoothness structure. Our results could lead to new decentralized algorithms for multi-agent optimization with graph topology invariant gradient/sampling complexity and new ADMM algorithms for solving total variation image reconstruction problems with accelerated gradient complexity.

 

Application of NNLCIs to the scattering of electromagnetic waves around curved PECs

Series
Applied and Computational Mathematics Seminar
Time
Monday, March 27, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Hwi LeeGeorgia Tech Math

In this talk, we demonstrate the application of Neural Networks with Locally Converging Inputs (NNLCI) to simulate the scattering of electromagnetic waves around two-dimensional perfect electric conductors (PEC). The NNLCIs are designed to output high-fidelity numerical solutions from local patches of two coarse grid numerical solutions obtained by a convergent numerical scheme. Once trained, the NNLCIs can play the role of a computational cost-saving tool for repetitive computations with varying parameters. To generate the inputs to our NNLCI, we design on uniform rectangular grids a second-order accurate finite difference scheme that can handle curved PEC boundaries systematically. More specifically, our numerical scheme is based on the Back and Forth Error Compensation and Correction method together with the construction of ghost points via a level set framework, PDE-based extension technique, and what we term guest values. We illustrate the performance of our NNLCI subject to variations in incident waves as well as PEC boundary geometries.

The Surprising Robustness and Computational Efficiency of Weak Form System Identification

Series
Applied and Computational Mathematics Seminar
Time
Monday, March 13, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 (ZOOM)
Speaker
David BortzUniversity of Colorado, Boulder

Recent advances in data-driven modeling approaches have proven highly successful in a wide range of fields in science and engineering. In this talk, I will briefly discuss several ubiquitous challenges with the conventional model development / discretization / parameter inference / model revision loop that our methodology attempts to address. I will present our weak form methodology which has proven to have surprising performance properties. In particular, I will describe our equation learning (WSINDy) and parameter estimation (WENDy) algorithms.  Lastly, I will discuss applications to several benchmark problems illustrating how our approach addresses several of the above issues and offers advantages in terms of computational efficiency, noise robustness, and modest data needs (in an online learning context).

Optimal Transport for Averaged Control

Series
Applied and Computational Mathematics Seminar
Time
Monday, March 6, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skile 005 and https://gatech.zoom.us/j/98355006347
Speaker
Dr. Daniel Owusu AduUGA

We study the problem of designing a robust parameter-independent feedback control input that steers, with minimum energy, the average of a linear system submitted to parameter perturbations where the states are initialized and finalized according to a given initial and final distribution. We formulate this problem as an optimal transport problem, where the transport cost of an initial and final state is the minimum energy of the ensemble of linear systems that have started from the initial state and the average of the ensemble of states at the final time is the final state. The by-product of this formulation is that using tools from optimal transport, we are able to design a robust parameter-independent feedback control with minimum energy for the ensemble of uncertain linear systems. This relies on the existence of a transport map which we characterize as the gradient of a convex function.

Generalization and sampling from the dynamics perspective

Series
Applied and Computational Mathematics Seminar
Time
Monday, February 27, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Prof. Nisha ChandramoorthyGT CSE

Please Note: Speaker will present in person

In this talk, we obtain new computational insights into two classical areas of statistics: generalization and sampling. In the first part, we study generalization: the performance of a learning algorithm on unseen data. We define a notion of generalization for non-converging training with local descent approaches via the stability of loss statistics. This notion yields generalization bounds in a similar manner to classical algorithmic stability. Then, we show that more information from the training dynamics provides clues to generalization performance.   

In the second part, we discuss a new method for constructing transport maps. Transport maps are transformations between the sample space of a source (which is generally easy to sample) and a target (typically non-Gaussian) probability distribution. The new construction arises from an infinite-dimensional generalization of a Newton method to find the zero of a "score operator". We define such a score operator that gives the difference of the score -- gradient of logarithm of density -- of a transported distribution from the target score. The new construction is iterative, enjoys fast convergence under smoothness assumptions, and does not make a parametric ansatz on the transport map.

Scalable Bayesian optimal experimental design for efficient data acquisition

Series
Applied and Computational Mathematics Seminar
Time
Monday, February 20, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Peng ChenGeorgia Tech CSE

Bayesian optimal experimental design (OED) is a principled framework for maximizing information gained from limited data in Bayesian inverse problems. Unfortunately, conventional methods for OED are prohibitive when applied to expensive models with high-dimensional parameters. In this talk, I will present fast and scalable computational methods for large-scale Bayesian OED with infinite-dimensional parameters, including data-informed low-rank approximation, efficient offline-online decomposition, projected neural network approximation, and a new swapping greedy algorithm for combinatorial optimization.

 

Embedded solitary internal waves

Series
Applied and Computational Mathematics Seminar
Time
Monday, February 13, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005
Speaker
Paul MilewskiUniversity of Bath, UK/Courant Institute NYU

Please Note: We expect to have an online option available: https://gatech.zoom.us/j/98355006347

The ocean and atmosphere are density stratified fluids. A wide variety of gravity waves propagate in their interior, redistributing energy and mixing the fluid, affecting global climate balances.  Stratified fluids with narrow regions of rapid density variation with respect to depth (pycnoclines) are often modelled as layered flows. We shall adopt this model and examine horizontally propagating internal waves within a three-layer fluid, with a focus on mode-2 waves which have oscillatory vertical structure and whose observations and modelling have only recently started. Mode-2 waves (typically) occur within the linear spectrum of mode-1 waves (i.e. they travel at lower speeds than mode-1 waves), and thus mode-2 solitary waves are  generically associated with an unphysical resonant mode-1 infinite oscillatory tail. We will show that these tail oscillations can be found to have zero amplitude, thus resulting in families of localised solutions (so called embedded solitary waves) in the Euler equations. This is the first example we know of embedded solitary waves in the Euler equations, and would imply that these waves are longer lived that previously thought.

Pages