Ilija Radosavovic – Robotics as Sensorimotor Sequence  Modeling

Abstract:

Over the last decade, large language models trained by next word prediction  
have provided a unified framework for natural language processing tasks. In  
this talk, I will demonstrate how the same paradigm, when sufficiently  
generalized, can provide an effective approach to robotics. As a  
“language” for robotics, we use sequences of sensory observations and  
motor commands, interleaved over time. This sensorimotor sequence modeling  
approach enables tackling multiple robotic tasks. In locomotion, it enables  
learning humanoid locomotion over challenging terrain, including hiking in  
the Berkeley Hills and climbing the steepest streets in San  Francisco achieved through next token prediction pre-training followed by  reinforcement learning fine-tuning. In manipulation, the same approach enables bimanual dexterous manipulation from pixels.
Beyond these capabilities, I will discuss how adaptive behaviors and capabilities.
I will discuss how adaptive behaviors and rich representations emerge as a byproduct of learning. 

Bio:

Ilija Radosavovic is a Ph.D. student in EECS at UC Berkeley, advised by  
Professor Jitendra Malik. His research interests are in the areas of  
robotics, computer vision, and machine learning. Ilija is a recipient of the  
PAMI Mark Everingham Award (2021), and his work has been deployed across  
industry and adopted by major corporations, including Facebook and Tesla.

Details

• Date: Thursday, April 17
• Time: 11:00 am - 12:00 pm
• Category: EECS Seminar
• Location: 34-401 Grier A

Host

• Leslie P Kaelbling