MIT Department of Electrical Engineering & Computer Science

Sensory-Grounded Language Learning

Deb Roy
MIT Media Laboratory

Wednesday, April 21, 1999
4:15 PM (refreshments 4:00)
Room NE43-941
EECS Special Seminar

Abstract

I will present a computational model of early language learning which acquires a lexicon from raw auditory and visual input. The model uses a multidirectional inference process in which acoustic events and co-occurring visual observations serve as mutual constraints for learning to segment speech, form visual categories, and ultimately establish associations between spoken words and their visual semantics.

The model has been implemented on a custom-built robotic platform. The system gathers images of objects taken from multiple perspectives with a CCD camera that is mounted on the robot. Acoustic and visual analysis methods have been developed to extract robust representations from microphone and camera input. A novel on-line algorithm operates on these audio-visual representations to learn a sensory-grounded lexicon. The acquired lexicon can be used to understand and generate spoken words. The current system learns color and shape terms by interacting with a person who presents the robot with objects while talking about them.

This research is motivated by two complementary goals. The first goal is to develop a cognitively plausible model of early word learning which makes use of multimodal sensory input. A study of parent-infant interactions during the early word learning phase has been undertaken to evaluate the model in this regard. I will present results of using recordings from these experiments as input to the model.

A second motivation is to develop human-computer interfaces which can tune to the language usage patterns of individuals. The goal is to create an adaptive and natural interface which learns to understand the user, regardless of individual differences in acoustic and semantic form. I will present on-going efforts to integrate the word learning system into various application domains including entertainment, education, command-and-control, and assistive aids.

HOSTS: Professor R. Brooks and Professor T. Lozano-Perez