Doctoral Thesis: Steering Robots with Inference-Time Interactions

By: Felix Wang

Thesis Supervisor: Julie Shah

Committee Members: Leslie Kaelbling, Jacob Andreas, Dorsa Sadigh

Details

• Date: Tuesday, February 25
• Time: 12:00 pm - 2:00 pm
• Category: Thesis Defense
• Location: Location if in person: 45-792 Zoom link: contact felixw@mit.edu

Additional Location Details:

Abstract: Imitation learning has driven the development of generalist policies capable of autonomously solving multiple tasks. However, when a pretrained policy makes errors during deployment, there are limited mechanisms for users to steer its behavior. While collecting additional data for fine-tuning can address such issues, doing so for each downstream use case is inefficient at scale. My research proposes an alternative perspective: framing policy errors as task mis-specifications rather than skill deficiencies. By enabling users to specify tasks unambiguously via interactions at inference-time, the appropriate skill for a given context can be retrieved without fine-tuning. Specifically, I propose (1) inference-time steering, which leverages human interactions for single-step task specification, and (2) task and motion imitation, which uses symbolic plans for multi-step task specification. These frameworks correct misaligned policy predictions without requiring additional training, maximizing the utility of pretrained models while achieving inference-time user objectives.

Host

• Felix Wang
• Email: felixw@mit.edu