Mean-Field Dynamics in Neural Networks: Implications for Generalization and Applications
Abstract:
The surprising flexibility and undeniable empirical success of machine learning algorithms has inspired many theoretical explanations for the efficacy of neural networks. Here, I will briefly introduce one perspective that provides not only asymptotic guarantees of trainability and accuracy in high-dimensional learning problems but also provides some prescriptions and design principles for learning. Bolstered by the favorable scaling of these algorithms in high dimensional problems, I will turn to the problem of variational high dimensional PDEs. From the perspective of an applied mathematician, these problems typically appear hopeless; they are not only high-dimensional but also dominated by rare events. However, with neural networks in the toolkit, at least the dimensionality is somewhat less intimidating. I will describe an algorithm that combines stochastic gradient descent with importance sampling to optimize a function representation of the solution. Finally, I will provide numerical evidence of the power and limitations of this approach.
Bio:
Grant Rotskoff studies the nonequilibrium dynamics of living matter with a particular focus on self-organization from the molecular to the cellular scale. His work involves developing theoretical and computational tools that can probe and predict the properties of physical systems driven away from equilibrium. Recently, he has focused on characterizing and designing physically accurate machine learning techniques for biophysical modeling. Prior to his current position, Grant was a James S. McDonnell Fellow working at the Courant Institute of Mathematical Sciences at New York University. He completed his Ph.D. at the University of California, Berkeley in the Biophysics graduate group supported by an NSF Graduate Research Fellowship. His thesis, which was advised by Phillip Geissler and Gavin Crooks, developed theoretical tools for understanding non-equilibrium control of the small, fluctuating systems, such as those encountered in molecular biophysics. He also worked on coarse-grained models of the hydrophobic effect and self-assembly. Grant received an S.B. in Mathematics from the University of Chicago, where he became interested in biophysics as an undergraduate while working on free energy methods for large-scale molecular dynamics simulations.