Georgia Institute of Technology College of Sciences

For a graphic degree sequence $\mathbf{d_n}= (d_1 , . . . , d_n)$ of graphs with vertices $v_1 , . . . , v_n$, $\mathbf{d_n}$-process is the random graph process that inserts one edge at a time at random with the restriction that the degree of $v_i$ is at most $d_i$ . In 1999, N. Wormald asked whether the final graph of random $\mathbf{d_n}$-process is "similar" to the uniform random graph with degree sequence $\mathbf{d_n}$ when $\mathbf{d_n}=(d,\dots, d)$. We answer this question for the $\mathbf{d_n}$-process when the degree sequence $\mathbf{d_n}$ that is not close to being regular. We used the method of switching for stochastic processes; this allows us to track the edge statistics of the $\mathbf{d_n}$-process. Joint work with Mike Molloy and Lutz Warnke.

Large-scale machine learning models are trained by parallel (stochastic) gradient descent algorithms on distributed systems. The communications for gradient aggregation and model synchronization become the major obstacles for efficient learning as the number of nodes and the model's dimension scale up. In this talk, I will introduce several ways to compress the transferred data and reduce the overall communication such that the obstacles can be immensely mitigated. More specifically, I will introduce methods to reduce or eliminate the compression error without additional communication.

We review some theoretical and computational results on locating eigenvalues coalescence for matrices smoothly depending on parameters. Focus is on the symmetric 2 parameter case, and Hermitian 3 parameter case. Full and banded matrices are of interest.

This talk will report some of our progress in showing how dynamics can be a useful mathematical tool for machine learning. Three demonstrations will be given, namely, how dynamics help design (and analyze) optimization algorithms, how dynamics help quantitatively understand nontrivial observations in deep learning practices, and how deep learning can in turn help dynamics (or more broadly put, AI for sciences). More precisely, in part 1 (dynamics for algorithm): I will talk about how to add momentum to gradient descent on a class of manifolds known as Lie groups. The treatment will be based on geometric mechanics and an interplay between continuous and discrete time dynamics. It will lead to accelerated optimization. Part 2 (dynamics for understanding deep learning) will be devoted to better understanding the nontrivial effects of large learning rates. I will describe how large learning rates could deterministically lead to chaotic escapes from local minima, which is an alternative mechanism to commonly known noisy escapes due to stochastic gradients. I will also mention another example, on an implicit regularization effect of large learning rates which is to favor flatter minimizers.  Part 3 (AI for sciences) will be on data-driven prediction of mechanical dynamics, for which I will demonstrate one strong benefit of having physics hard-wired into deep learning models; more precisely, how to make symplectic predictions, and how that generically improves the accuracy of long-time predictions.