Georgia Institute of Technology College of Sciences

Artificial neural networks have gained widespread adoption as a powerful tool for various machine learning tasks in recent years. Training a neural network to approximate a target function involves solving an inherently non-convex problem. In practice, this is done using stochastic gradient descent with random initialization. For the approximation problem with neural networks error rate guarantees are established for different classes of functions however these rates are not always achieved in practice due to many  local minima of the resulting optimization problem. 

The challenge we address in this work is the following. We want to find small size shallow neural networks that can be trained algorithmically and which achieve guaranteed approximation speed and precision. To maintain the small size we apply penalties on the weights of the network. We show that under minimal requirements, all local minima of the resulting problem are well behaved and possess a desirable small size without sacrificing precision. We adopt the integral neural network framework and use techniques from optimization theory and harmonic analysis to prove our results. In this talk, we will discuss our existing work and possible future promising areas of interest where this approach can potentially be adopted. 

Tba

 The problem of phase retrieval for a set of functions $H$ can be thought of as being able to identify a function $f\in H$ or $-f\in H$ from the absolute value $|f|$.  Phase retrieval for a set of functions is called stable if when $|f|$ and $|g|$ are close then $f$ is proportionally close to $g$ or $-g$.  That is, we say that a set $H\subseteq L_2({\mathbb R})$ does stable phase retrieval if there exists a constant $C>0$ so that
$$\min\big(\big\|f-g\big\|_{L_2({\mathbb R})},\big\|f+g\big\|_{L_2({\mathbb R})}\big)\leq C \big\| |f|-|g| \big\|_{L_2({\mathbb R})} \qquad\textrm{ for all }f,g\in H.$$
 It is known that phase retrieval for finite dimensional spaces is always stable.  On the other hand, phase retrieval for infinite dimensional spaces using a frame or a continuous frame is always unstable.  We prove that there exist infinite dimensional subspaces of $L_2({\mathbb R})$ which do stable phase retrieval.  This is joint work with Robert Calderbank, Ingrid Daubechies, and Nikki Freeman.