Please Note: This seminar will be given online.
|
Probability seminar series Mihai Nica Link to join seminar: Hosted on Webex |
Log-normal Behavior in Deep Neural Nets and Products of Random Matrices
Deep neural networks (DNNs) have achieved tremendous results in many domains. However, the theoretical mathematical understanding of these systems is still in its infancy. One way to understand neural networks is by looking in the limit where the number of parameters tends to infinity. For a network of fixed depth, it has been shown that DNNs can be understand as Gaussian processes. In contrast to this Gaussian regime, I will present recent work that shows that very deep networks instead have log-normal behavior. The relevant limit now sends both the width AND depth of the network to infinity simultaneously; the log-normal behavior depends on ratio of depth to width. The result comes from analyzing products of random matrices when both the size of the matrices and the number of elements in the product tend to infinity simultaneously. Based on https://arxiv.org/abs/1812.05994 and https://arxiv.org/abs/1909.05989