AI & Fundamentals

Using Algorithms to Understand Transformers (and Using Transformers to Understand Algorithms) - Vatsal Sharan, Assistant Professor, University of Southern California

Abstract:

We will discuss how algorithmic tools and understanding borrowed from optimization theory, Fourier transforms, and Boolean function analysis can help understand the mechanisms employed by Transformers to solve basic computational tasks such as linear regression and addition. We will examine the role of the architecture and pre-trained data in enabling Transformers to learn their employed mechanisms. Finally, we will discuss work on using Transformers themselves to discover and design data structures for tasks such as nearest neighbor search.
 

Bio

Vatsal Sharan is an assistant professor of computer science at the University of Southern California (USC), and works on the foundations of machine learning. Previously, he did his undergraduate at IIT Kanpur, PhD at Stanford and a postdoc at MIT. His work has been recognized with the best paper award at COLT, NSF Career award and Amazon research award.