Georgia Institute of Technology College of Sciences

In this talk I will present some of their theoretical guarantees with an emphasis on their behavior under the so-called manifold hypothesis. Such theoretical guarantees are non-vacuous and provide insight on the empirical behavior of these models. I will show how these results imply generalization bounds on denoising diffusion models. This presentation is based on https://arxiv.org/abs/2208.05314

MCMC and variational inference are two competing paradigms for the problem of sampling from a given probability distribution. In this talk, I'll show how they can work together to give the first polynomial-time sampling algorithm for approximately low-rank Ising models. Sampling was previously known when all eigenvalues of the interaction matrix fit in an interval of length 1; however, a single outlier can cause Glauber dynamics to mix torpidly. Our result covers the case when all but O(1) eigenvalues lie in an interval of length 1. To deal with positive eigenvalues, we use a temperature-based heuristic for MCMC called simulated tempering, while to deal with negative eigenvalues, we define a nonconvex variational problem over Ising models, solved using SGD. Our result has applications to sampling Hopfield networks with a fixed number of patterns, Bayesian clustering models with low-dimensional contexts, and antiferromagnetic/ferromagnetic Ising model on expander graphs.

Given a collection of functions in a Banach space, typically called a dictionary in machine learning, we study the approximation properties of its convex hull. Specifically, we develop techniques for bounding the metric entropy and n-widths, which are fundamental quantities in approximation theory that control the limits of linear and non-linear approximation. Our results generalize existing methods by taking the smoothness of the dictionary into account, and in particular give sharp estimates for shallow neural networks. Consequences of these results include: the optimal approximation rates which can be attained for shallow neural networks, that shallow neural networks dramatically outperform linear methods of approximation, and indeed that shallow neural networks outperform all continuous methods of approximation on the associated convex hull. Next, we discuss greedy algorithms for constructing approximations by non-linear dictionary expansions. Specifically, we give sharp rates for the orthogonal greedy algorithm for dictionaries with small metric entropy, and for the pure greedy algorithm. Finally, we give numerical examples showing that greedy algorithms can be used to solve PDEs with shallow neural networks.

In the past decade, the revival of deep neural networks has led to dramatic success in numerous applications ranging from computer vision to natural language processing to scientific discovery and beyond. Nevertheless, the practice of deep networks has been shrouded with mystery as our theoretical understanding of the success of deep learning remains elusive.

In this talk, we will exploit low-dimensional modeling to help understand and improve deep learning performance. We will first provide a geometric analysis for understanding neural collapse, an intriguing empirical phenomenon that persists across different neural network architectures and a variety of standard datasets. We will utilize our understanding of neural collapse to improve training efficiency. We will then exploit principled methods for dealing with sparsity and sparse corruptions to address the challenges of overfitting for modern deep networks in the presence of training data corruptions. We will introduce a principled approach for robustly training deep networks with noisy labels and robustly recovering natural images by deep image prior.