Georgia Institute of Technology College of Sciences

Pretrained Transformers adapt to an unseen task from a short prompt, with no parameter update. This ability, known as in-context learning (ICL), raises a basic question: how does the structure shared across tasks reduce the number of examples a new task requires? We quantify low-dimensional cross-task structure through the covering numbers of the task space, without assuming an explicit parametric representation. The cover supplies anchor functions, and ICL reduces to a task-identification-and-evaluation procedure: the context localizes the unseen task among the anchors, and the prediction at a query aggregates the anchors’ values there. We construct a three-block Transformer that implements this procedure and derive a generalization bound that separates the number of pretraining tasks from the prompt length: the pretraining cost is governed by the intrinsic dimensions of the task space and input domain, while, with sufficiently many pretraining tasks, the dependence on the prompt length becomes dimension-free. The key technical tool is a local-to-global approximation principle realized by a softmax partition of unity: with suitably chosen affine scores, softmax attention produces localized weights that sum to one, so that dense, shallow Transformers with standard softmax attention and sinusoidal positional encodings can glue local approximations into a global one. We further extend this idea to operator learning: a two-level softmax partition of unity, over the input function class and the output domain, yields power-law model- and data-scaling laws for Transformer operator networks on input function classes of low intrinsic dimension. Experiments on the Burgers’ and KdV equations confirm the predicted scaling.