Seminars and Colloquia by Series

Uncovering the Law of Data Separation in Deep Learning

Series
Applied and Computational Mathematics Seminar
Time
Monday, April 17, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Prof. Weijie Su – University of Pennsylvania (Wharton)

Please Note: The speaker will present in person.

In this talk, we will investigate the emergence of geometric patterns in well-trained deep learning models by making use of a layer-peeled model and the law of equi-separation. The former is a nonconvex optimization program that models the last-layer features and weights. We use the model to shed light on the neural collapse phenomenon of Papyan, Han, and Donoho, and to predict a hitherto-unknown phenomenon that we term minority collapse in imbalanced training.
 
The law of equi-separation is a pervasive empirical phenomenon that describes how data are separated according to their class membership from the bottom to the top layer in a well-trained neural network. We will show that, through extensive computational experiments, neural networks improve data separation through layers in a simple exponential manner. This law leads to roughly equal ratios of separation that a single layer is able to improve, thereby showing that all layers are created equal. We will conclude the talk by discussing the implications of this law on the interpretation, robustness, and generalization of deep learning, as well as on the inadequacy of some existing approaches toward demystifying deep learning.
 

Mathematical Foundations of Graph-Based Bayesian Semi-Supervised Learning

Series
Applied and Computational Mathematics Seminar
Time
Monday, April 10, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Prof. Daniel Sanz-Alonso – U Chicago

Please Note: Speaker will present in person

Semi-supervised learning refers to the problem of recovering an input-output map using many unlabeled examples and a few labeled ones. In this talk I will survey several mathematical questions arising from the Bayesian formulation of graph-based semi-supervised learning. These questions include the modeling of prior distributions for functions on graphs, the derivation of continuum limits for the posterior, the design of scalable posterior sampling algorithms, and the contraction of the posterior in the large data limit.

New gradient sliding results on convex optimization with smoothness structure

Series
Applied and Computational Mathematics Seminar
Time
Monday, April 3, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Yuyuan Ouyang – Clemson University –

In this talk, we present new gradient sliding results for constrained convex optimization with applications in image reconstruction and decentralized distributed optimization. Specifically, we will study classes of large-scale problems that minimizes a convex objective function over feasible set with linear constraints. We will show that by exploring the gradient sliding technique, the number of gradient evaluations of the objective function can be reduced by exploring the smoothness structure. Our results could lead to new decentralized algorithms for multi-agent optimization with graph topology invariant gradient/sampling complexity and new ADMM algorithms for solving total variation image reconstruction problems with accelerated gradient complexity.

 

Application of NNLCIs to the scattering of electromagnetic waves around curved PECs

Series
Applied and Computational Mathematics Seminar
Time
Monday, March 27, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Hwi Lee – Georgia Tech Math –

In this talk, we demonstrate the application of Neural Networks with Locally Converging Inputs (NNLCI) to simulate the scattering of electromagnetic waves around two-dimensional perfect electric conductors (PEC). The NNLCIs are designed to output high-fidelity numerical solutions from local patches of two coarse grid numerical solutions obtained by a convergent numerical scheme. Once trained, the NNLCIs can play the role of a computational cost-saving tool for repetitive computations with varying parameters. To generate the inputs to our NNLCI, we design on uniform rectangular grids a second-order accurate finite difference scheme that can handle curved PEC boundaries systematically. More specifically, our numerical scheme is based on the Back and Forth Error Compensation and Correction method together with the construction of ghost points via a level set framework, PDE-based extension technique, and what we term guest values. We illustrate the performance of our NNLCI subject to variations in incident waves as well as PEC boundary geometries.

The Surprising Robustness and Computational Efficiency of Weak Form System Identification

Series
Applied and Computational Mathematics Seminar
Time
Monday, March 13, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 (ZOOM)
Speaker
David Bortz – University of Colorado, Boulder –

Recent advances in data-driven modeling approaches have proven highly successful in a wide range of fields in science and engineering. In this talk, I will briefly discuss several ubiquitous challenges with the conventional model development / discretization / parameter inference / model revision loop that our methodology attempts to address. I will present our weak form methodology which has proven to have surprising performance properties. In particular, I will describe our equation learning (WSINDy) and parameter estimation (WENDy) algorithms.  Lastly, I will discuss applications to several benchmark problems illustrating how our approach addresses several of the above issues and offers advantages in terms of computational efficiency, noise robustness, and modest data needs (in an online learning context).

Optimal Transport for Averaged Control

Series
Applied and Computational Mathematics Seminar
Time
Monday, March 6, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skile 005 and https://gatech.zoom.us/j/98355006347
Speaker
Dr. Daniel Owusu Adu – UGA

We study the problem of designing a robust parameter-independent feedback control input that steers, with minimum energy, the average of a linear system submitted to parameter perturbations where the states are initialized and finalized according to a given initial and final distribution. We formulate this problem as an optimal transport problem, where the transport cost of an initial and final state is the minimum energy of the ensemble of linear systems that have started from the initial state and the average of the ensemble of states at the final time is the final state. The by-product of this formulation is that using tools from optimal transport, we are able to design a robust parameter-independent feedback control with minimum energy for the ensemble of uncertain linear systems. This relies on the existence of a transport map which we characterize as the gradient of a convex function.

Generalization and sampling from the dynamics perspective

Series
Applied and Computational Mathematics Seminar
Time
Monday, February 27, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Prof. Nisha Chandramoorthy – GT CSE

Please Note: Speaker will present in person

In this talk, we obtain new computational insights into two classical areas of statistics: generalization and sampling. In the first part, we study generalization: the performance of a learning algorithm on unseen data. We define a notion of generalization for non-converging training with local descent approaches via the stability of loss statistics. This notion yields generalization bounds in a similar manner to classical algorithmic stability. Then, we show that more information from the training dynamics provides clues to generalization performance.   

In the second part, we discuss a new method for constructing transport maps. Transport maps are transformations between the sample space of a source (which is generally easy to sample) and a target (typically non-Gaussian) probability distribution. The new construction arises from an infinite-dimensional generalization of a Newton method to find the zero of a "score operator". We define such a score operator that gives the difference of the score -- gradient of logarithm of density -- of a transported distribution from the target score. The new construction is iterative, enjoys fast convergence under smoothness assumptions, and does not make a parametric ansatz on the transport map.

Scalable Bayesian optimal experimental design for efficient data acquisition

Series
Applied and Computational Mathematics Seminar
Time
Monday, February 20, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005 and https://gatech.zoom.us/j/98355006347
Speaker
Peng Chen – Georgia Tech CSE –

Bayesian optimal experimental design (OED) is a principled framework for maximizing information gained from limited data in Bayesian inverse problems. Unfortunately, conventional methods for OED are prohibitive when applied to expensive models with high-dimensional parameters. In this talk, I will present fast and scalable computational methods for large-scale Bayesian OED with infinite-dimensional parameters, including data-informed low-rank approximation, efficient offline-online decomposition, projected neural network approximation, and a new swapping greedy algorithm for combinatorial optimization.

 

Embedded solitary internal waves

Series
Applied and Computational Mathematics Seminar
Time
Monday, February 13, 2023 - 14:00 for 1 hour (actually 50 minutes)
Location
Skiles 005
Speaker
Paul Milewski – University of Bath, UK/Courant Institute NYU –

Please Note: We expect to have an online option available: https://gatech.zoom.us/j/98355006347

The ocean and atmosphere are density stratified fluids. A wide variety of gravity waves propagate in their interior, redistributing energy and mixing the fluid, affecting global climate balances.  Stratified fluids with narrow regions of rapid density variation with respect to depth (pycnoclines) are often modelled as layered flows. We shall adopt this model and examine horizontally propagating internal waves within a three-layer fluid, with a focus on mode-2 waves which have oscillatory vertical structure and whose observations and modelling have only recently started. Mode-2 waves (typically) occur within the linear spectrum of mode-1 waves (i.e. they travel at lower speeds than mode-1 waves), and thus mode-2 solitary waves are  generically associated with an unphysical resonant mode-1 infinite oscillatory tail. We will show that these tail oscillations can be found to have zero amplitude, thus resulting in families of localised solutions (so called embedded solitary waves) in the Euler equations. This is the first example we know of embedded solitary waves in the Euler equations, and would imply that these waves are longer lived that previously thought.

Effective deep neural network architectures for learning high-dimensional Banach-valued functions from limited data

Series
Applied and Computational Mathematics Seminar
Time
Friday, February 10, 2023 - 11:00 for 1 hour (actually 50 minutes)
Location
Skiles 006 and https://gatech.zoom.us/j/98355006347
Speaker
Nick Dexter – Florida State University –

In the past few decades the problem of reconstructing high-dimensional functions taking values in abstract spaces from limited samples has received increasing attention, largely due to its relevance to uncertainty quantification (UQ) for computational science and engineering. These UQ problems are often posed in terms of parameterized partial differential equations whose solutions take values in Hilbert or Banach spaces. Impressive results have been achieved on such problems with deep learning (DL), i.e. machine learning with deep neural networks (DNN). This work focuses on approximating high-dimensional smooth functions taking values in reflexive and typically infinite-dimensional Banach spaces. Our novel approach to this problem is fully algorithmic, combining DL, compressed sensing, orthogonal polynomials, and finite element discretization. We present a full theoretical analysis for DNN approximation with explicit guarantees on the error and sample complexity, and a clear accounting of all sources of error. We also provide numerical experiments demonstrating the efficiency of DL at approximating such high-dimensional functions from limited data in UQ applications.
 

Pages