AI & Fundamentals
Ensembles in the Age of Overparameterization: Promises and Pathologies - Geoff Pleiss, Assistant Professor, UBC
DATE: Wed, October 23, 2024 - 11:00 am
LOCATION: UBC Vancouver Campus, ICCS X836 / Please register to receive Zoom link
DETAILS
Abstract:
Ensemble methods have historically used either high-bias base learners (e.g. through boosting) or high-variance base learners (e.g. through bagging). Modern neural networks cannot be understood through this classic bias-variance tradeoff, yet "deep ensembles" are pervasive in safety-critical and high-uncertainty application domains. This talk will cover surprising and counterintuitive phenomena that emerge when ensembling overparameterized base models like neural networks. While deep ensembles improve generalization in a simple and cost-effective manner, their accuracy and robustness are often outperformed by single (but larger) models. Furthermore, discouraging diversity amongst component models often improves the ensemble's predictive performance, counter to classic intuitions underpinning bagging and feature subsetting techniques. I will connect these empirical findings with new theoretical characterizations of overparameterized ensembles, and I will conclude with implications for uncertainty quantification, robustness, and decision making.
Bio
Dr. Geoff Pleiss is an assistant professor in the Department of Statistics at the University of British Columbia, where he is an inaugural member of CAIDA's AIM-SI (AI Methods for Scientific Impact) cluster. He is also a Canada CIFAR AI Chair and a faculty member at the Vector Institute. Dr. Geoff Pleiss' research interests intersect deep learning and probabilistic modeling. More specifically, he's interested in heuristic and approximate notions of uncertainty from machine learning models, and how they can inform reliable and optimal downstream decisions within the contexts of experimental design and scientific discovery. Major focuses of his work include: neural network uncertainty quantification, Bayesian optimization, Gaussian processes, and ensemble methods.
Geoff Pleiss is also an active open source contributor. Most notably, he co-created and maintains the GPyTorch Gaussian process library with Jake Gardner. Previously, he was a postdoc at Columbia University with John P. Cunningham. He received his Ph.D. from the CS department at Cornell University in 2020 where he was advised by Kilian Weinberger and also worked closely with Andrew Gordon Wilson.