Seminar

CVL Talk: Biologically Plausible Learning Using Local Activity Perturbation - Mengye Ren, NYU

Abstract:

Backprop is usually considered biologically implausible due to the issues of weight transport and global synchronization. Perturbation learning is potentially a good candidate for a biologically plausible alternative, since it does not require explicit backward weights, and recent advances in local greedy learning has suggested that deep networks can be trained well without global end-to-end learning. Forward-mode automatic differentiation can also be used to compute the effect of perturbation and can be viewed as the time derivative of an analog signal. These tools have shown us new opportunities for the search of a biologically plausible alternative to backprop.

Standard perturbation algorithms, however, suffer from the curse of dimensionality in the number of parameters. In this talk, I will introduce a series of architectural and algorithmic modifications that together make perturbation learning practical for standard deep learning benchmark tasks. We show that it is possible to substantially reduce the variance of the forward automatic derivative gradient estimator by applying perturbations to activations rather than weights. We further improve the scalability of forward gradient by introducing a large number of local greedy loss functions, each of which involves only a small number of learnable parameters, and a new MLPMixer-inspired architecture, LocalMixer, that is more suitable for local learning. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.

Bio:

Mengye Ren is an assistant professor of computer science and data science at New York University (NYU). Before joining NYU, he was a visiting faculty researcher at Google Brain Toronto working with Prof. Geoffrey Hinton. He received B.A.Sc. in Engineering Science (2015), and M.Sc. (2017) and Ph.D. (2021) in Computer Science from the University of Toronto, advised by Prof. Richard Zemel and Prof. Raquel Urtasun. From 2017 to 2021, he was also a senior research scientist at Uber Advanced Technologies Group (ATG) and Waabi, working on self-driving vehicles. His research focuses on making machine learning more natural and human-like, in order for AIs to continually learn, adapt, and reason in naturalistic environments.

Zoom Link

You can see the other CVPR 2023 UBC events here.