Mathematical Foundations of Neural Network Representation: Architecture, Regularization, and Function-Space Theories

Series
Applied and Computational Mathematics Seminar
Time
Monday, November 23, 2026 - 2:00pm for 1 hour (actually 50 minutes)
Location
Speaker
Julia Blair Nakhleh – University of Wisconsin, Madison – jnakhleh@wisc.edu – https://julianakhleh.github.io/
Organizer
Wenjing Liao

Neural networks are the workhorses of modern AI systems, yet our mathematical understanding of what kinds of functions they represent remains limited. To address this question, it is often informative to view neural networks as vectors in infinite-dimensional function spaces, whose norms impose different notions of control on the network parameters. We can then ask: what analytic and geometric structures do different architectures and regularizers (encoded by a particular neural function space norm) favor or constrain? In this talk, I will illustrate several concrete insights offered by this function-space perspective. I will begin by discussing shallow ReLU networks, and show how variations in architecture and regularization can produce functions with strikingly different analytic structures, even when trained to fit identical data. I will then discuss recent work on the representational capabilities of deep networks, with arbitrary width and controlled layerwise path norms. I will show that the statistical complexities of norm-bounded classes in these spaces grow only mildly with depth; moreover, in some special settings, I will show that depth provides essentially no benefit to functional expressivity behond two hidden layers. These results also identify certain types functional structure that networks of any width or depth cannot represent with small norm. Finally, I will connect our function-space norm to several commonly studied parameter-based complexity measures, showing that some apparent expressivity advantages of depth disappear when complexity is measured using appropriate function-space norms. Overall, these function-space analyses help explain how architecture and regularization introduce structural biases into highly overparameterized networks, influencing both their representational capabilities and the functions they select when fitting data.