AI & Fundamentals

Reinforcement Learning With Constraints: From Theory to Reasoning in LLM - Lin Yang, Assistant Professor, UCLA

Abstract:

In this talk, I will explore reinforcement learning with constraints, focusing on both theoretical foundations and practical applications. I will first present recent advances in the sample complexity of constrained Markov decision processes (CMDPs), covering both offline and online settings. Our results establish near-optimal upper and lower bounds under relaxed and strict feasibility regimes, revealing that constraint satisfaction—while generally harder—can match the sample efficiency of unconstrained MDPs under certain conditions. These insights are grounded in primal-dual algorithms and generative model frameworks. Inspired by this theory, I will discuss how CMDPs can be applied to impose behavior in large language models (LLMs), such as controlling reasoning length or enforcing budgeted constraints during fine-tuning. By treating response generation as a CMDP and incorporating online dual updates, we show that LLMs can be optimized to meet constraints with minimal degradation in performance. 

Bio:

Dr. Lin Yang (杨林) is an Associate Professor in the Electrical and Computer Engineering and Computer Science Departments at UCLA. His research centers on the foundations of modern machine learning and data science, with a focus on fast algorithms with provable guarantees in areas such as reinforcement learning, large language model acceleration, non-convex optimization, and streaming algorithms. Dr. Yang received dual Ph.D. degrees in Computer Science and Physics & Astronomy from Johns Hopkins University and was a postdoctoral researcher at Princeton University. His honors include the Amazon Faculty Award, Simons Research Fellowship, Dean Robert H. Roy Fellowship, and the JHU MINDS Best Dissertation Award.