Georgia Institute of Technology College of Sciences

Designing effective positional encodings for graphs is key to building powerful graph transformers and enhancing message-passing graph neural networks’ expressive power. However, since there lacks a canonical order of nodes in the graph-structure data, the choice of positional encodings for graphs is often tricky. For example, Laplacian eigenmap is used as positional encodings in many works. However, it faces two fundamental challenges: (1) Non-uniqueness: there are many different eigen-decompositions of the same Laplacian, and (2) Instability: small perturbations to the Laplacian could result in completely different eigenvectors, leading to unpredictable changes in positional encoding. This is governed by the Davis-Kahan theorem, which further negatively impacts the model generalization. In this talk, we are to introduce some ideas on building stable positional encoding and show their benefits in model out-of-distribution generalization. The idea can be extended to some other types of node positional encodings. Finally, we evaluate the effectiveness of our method on molecular property prediction, link prediction, and out-of-distribution generalization tasks, finding improved generalization compared to existing positional encoding methods.

I will mainly talk about three papers:

1. Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning,NeurIPS20, Pan Li, Yanbang Wang, Hongwei Wang, Jure Leskovec

2. Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks, ICLR22 Haorui Wang, Haoteng Yin, Muhan Zhang, Pan Li

3. On the Stability of Expressive Positional Encodings for Graphs, ICLR24 Yinan Huang, William Lu, Joshua Robinson, Yu Yang, Muhan Zhang, Stefanie Jegelka, Pan Li

Many numerical methods have been developed in the past years for computing weak solutions (with shock waves) to nonlinear hyperbolic conservation laws. My research, specifically, concerns the design of well-balanced numerical algorithms that preserve certain key structure of these equations in various applications, including for problems involving moving phase boundaries and other scale-dependent interfaces. In particular, in this lecture, I will focus on the evolution of a compressible fluid in spherical symmetry on a Schwarzschild curved background, for which I have designed a class of well-balanced numerical algorithms up to third-order of accuracy. Both the relativistic Burgers-Schwarzschild model and the relativistic Euler-Schwarzschild model were considered, and the proposed numerical algorithm took advantage of the explicit or implicit forms available for the stationary solutions of these models. The schemes follow the finite volume methodology and preserve the stationary solutions and, most importantly, allow us to investigate the global asymptotic behavior of such flows and determine the asymptotic behavior of the mass density and velocity field of the fluid. Blog: philippelefloch.org

The sparse solution obtained from greedy-based optimization approach such as orthogonal matching pursuit can be very useful and have many applications in different directions. In this talk, I will present two research projects, one is about semi-supervised local clustering, and the other is about function approximation, which make use of the sparse solution technique. We will show that the target cluster can be effectively retrieved in the local clustering task and the curse of dimensionality can be overcome for a dense subclass of the space of continuous functions via Kolmogorov superposition theorem. Both the theoretical and numerical results will be discussed.

Geometry arises in myriad ways within machine learning and related areas. I this talk I will focus on settings where geometry helps us understand problems in machine learning, optimization, and sampling. For instance, when sampling from densities supported on a manifold, understanding geometry and the impact of curvature are crucial; surprisingly, progress on geometric sampling theory helps us understand certain generalization properties of SGD for deep-learning! Another fascinating viewpoint afforded by geometry is in non-convex optimization: geometry can either help us make training algorithms more practical (e.g., in deep learning), it can reveal tractability despite non-convexity (e.g., via geodesically convex optimization), or it can simply help us understand existing methods better (e.g., SGD, eigenvector computation, etc.).

Ultimately, I hope to offer the audience some insights into geometric thinking and share with them some new tools that help us design, understand, and analyze models and algorithms. To make the discussion concrete I will recall a few foundational results arising from our research, provide several examples, and note some open problems.

––
Bio: Suvrit Sra is a Alexander von Humboldt Professor of Artificial Intelligence at the Technical University of Munich (Germany), and and Associate Professor of EECS at MIT (USA), where he is also a member of the Laboratory for Information and Decision Systems (LIDS) and of the Institute for Data, Systems, and Society (IDSS). He obtained his PhD in Computer Science from the University of Texas at Austin. Before TUM & MIT, he was a Senior Research Scientist at the Max Planck Institute for Intelligent Systems, Tübingen, Germany. He has held visiting positions at UC Berkeley (EECS) and Carnegie Mellon University (Machine Learning Department) during 2013-2014. His research bridges mathematical topics such as differential geometry, matrix analysis, convex analysis, probability theory, and optimization with machine learning. He founded the OPT (Optimization for Machine Learning) series of workshops, held from OPT2008–2017 at the NeurIPS  conference. He has co-edited a book with the same name (MIT Press, 2011). He is also a co-founder and chief scientist of Pendulum, a global AI+logistics startup.