Georgia Institute of Technology College of Sciences

Transformer (Vaswani et al. 2017) architecture is a popular deep learning architecture that today comprises the foundation for most tasks in natural language processing and forms the backbone of all the current state-of-the-art language models. Central to its success is the attention mechanism, which allows the model to weigh the importance of different input tokens. However, Transformers can become computationally expensive, especially for large-scale tasks. To address this, researchers have explored techniques for conditional computation, which selectively activate parts of the model based on the input. In this talk, we present two case studies of conditional computation in Transformer models. In the first case, we examine the routing mechanism in the Mixture-of-Expert (MoE) Transformer models, and show theoretical and empirical evidence that the router’s ability to route intelligently confers a significant advantage to MoE models. In the second case, we introduce Alternating Updates (AltUp), a method to take advantage of increased residual stream width in the Transformer models without increasing the computation cost.

Speaker's brief introduction: Xin Wang is a research engineer in the Algorithms team at Google Research. Xin finished his PhD in Mathematics at Georgia Institute of Technology before coming to Google. Xin's research interests include efficient computing, memory mechanism for machine learning, and optimization.

The talk will be presented online at

 https://gatech.zoom.us/j/93087689904

The late Goro Shimura proposed a question regarding certain invariant differential operators on a Hermitian symmetric space. This was answered by Sahi and Zhang by showing that the Harish-Chandra images of these namesake operators are specializations of Okounkov's BC-symmetric interpolation polynomials. We prove, in the super setting, that the Harish-Chandra images of super Shimura operators are specializations of certain BC-supersymmetric interpolation polynomials due to Sergeev and Veselov. Similar questions include the Capelli eigenvalue problems which are generalized to the quantum and/or super settings. This talk is based a joint work with Siddhartha Sahi.

In the 1990s, Arkadi Nemirovski asked the following question:

How hard is it to estimate a solution to unknown homogeneous linear difference equation with constant coefficients of order S, observed in the Gaussian noise on [0,N]?

The class of all such solutions, or "signals," is parametric---described by 2S complex parameters---but extremely rich: it includes the weighted sums of S exponentials, polynomials of degree S, harmonic oscillations with S arbitrary frequencies, and their algebraic combinations. Geometrically, this class is the union of all S-dimensional shift-invariant subspaces of the space of two-sided sequences, and of interest is the minimax risk on it with respect to the mean-squared error on [0,N]. I will present a recent result that shows this minimax risk to be O( S log(N) log(S)^2 ), improving over the state of the art by a polynomial in S factor, and coming within an O( log(S)^2 ) factor from the lower bound. It relies upon an approximation-theoretic construction related to minimal-norm interpolation over shift-invariant subspaces, in the spirit of the Landau-Kolmogorov problem, that I shall present in some detail. Namely, we will see that any shift-invariant subspace admits a bounded-support reproducing kernel whose spectrum has nearly the smallest possible Lp-energies for all p ≥ 1 at once.

We say a class of graphs $\mathcal{F}$ is degree-bounded if there exists a function $f$ such that $\delta(G)\le f(\tau(G))$ for every $G\in\mathcal{F}$, where $\delta(G)$ denotes the minimum degree of $G$ and $\tau(G)$ is the biclique number of $G$, that is, the largest integer $t$ such that $G$ contains $K_{t,t}$ as a subgraph. Such a function $f$ is called a degree-bounding function for $\mathcal{F}$.

In joint work with Ant\'onio Gir\~ao, Zach Hunter, Rose McCarty and Alex Scott, we proved that every hereditary degree-bounded class $\mathcal{F}$ has a degree-bounding function that is singly exponential in the biclique number $\tau$. A more recent result by Ant\'onio Gir\~ao and Zach Hunter improved this bound to a polynomial function in $\tau$. In this talk, I will discuss examples and the recent results on degree-boundedness.