Georgia Institute of Technology College of Sciences

Many emerging applications of artificial intelligence, from analyzing scientific literature to developing complex software, rely on language models to process increasingly large amounts of information. Yet the ability to accept a long input, or context, does not guarantee that a model can use it effectively. In this talk, I will investigate how language models use long contexts through mathematical models of transformers, the architecture underlying many language models.

I will first study how language models learn from examples supplied in their input without changing their parameters, an ability known as in-context learning. A longer context can contain more examples, but how rapidly can a model benefit from this growing body of evidence? Using mean-field analysis, we establish quantitative guarantees for learning from finitely many examples.

The benefit of additional information, however, also depends on whether a model can focus on useful evidence amid irrelevant material. I will therefore study temperature scaling, a practical strategy that adjusts how selectively a model weighs different parts of its input. Drawing on extreme-value theory, we identify a critical scaling with context length at which only several selected pieces of information remain influential.

I will conclude by connecting these results to my broader research on mathematical machine learning and highlighting how applied mathematics can inform the design of reliable learning algorithms.