Doctoral Thesis: Realtime State Estimation for Contact Manipulation


Event Speaker: 

Peter Kuan-Ting Yu

Event Location: 


Event Date/Time: 

Friday, June 15, 2018 - 2:00pm


In this thesis, I develop a realtime state estimation system to recover the pose and contact status of an object relative to its environment. This capability is important for a controller to react to uncertainties in a manipulation task.

I propose a framework that fuses tactile and visual sensing to improve accuracy and robustness. Visual sensing is versatile and non-intrusive, but suffers from occlusions and limited accuracy, especially for tasks involving contact. Tactile sensing (including contact and force) is local, but provides accuracy and robustness to occlusions. I apply on-line optimization techniques to fuse kinematic measurements from the robot, contact geometry of the object and the environment, and visual measurements. In a complex contact task like insertion, the contact formations are complicated and hard to resolve directly. I propose a data-driven method to infer the contact formation, which is then used in realtime by the state estimator. I apply our framework to two iconic tasks in robotic manipulation: planar pushing and object insertion. I evaluate the algorithm in a setup instrumented to provide ground truth.


Peter is a PhD candidate of EECS. He received the degrees of B.S. in Computer Science from National Chiao-Tung University in 2010, and M.S. in Computer Science from National Taiwan University in 2012. He works with Prof. Alberto Rodriguez in the Manipulation & Mechanism lab. His research focuses on incorporating contact sensing and physics to improve robot perception. His paper on dataset collection for pushing manipulation was nominated for Best Paper Award in IROS 2016. He served as the perception and software lead in the MIT-Princeton Team, who won the Stowing task in the Amazon Robotics Challenge 2017. 


Alberto Rodriguez (Advisor)

Walter Henry Gale (1929) Career Development Professor


John Leonard

Samuel C. Collins Professor of Mechanical and Ocean Engineering 


Tomas Lozano-Perez

School of Engineering Professor of Teaching Excellence at MIT


Justin Solomon

X-Consortium Career Development Assistant Professor