Georgia Institute of Technology College of Sciences

To study the underlying mechanisms behind transformers and related techniques, we propose a transformer learning framework motivated by a two-stage sampling process, with distributions being inputs, and present a mathematical formulation of the attention mechanism as kernel embedding. Our findings show that by the attention operator, transformers can compress distributions into function representations without loss of information. We also demonstrate the in-context learning capabilities of efficient transformer structures through a rigorous generalization analysis.