Home Research Research Highlights
Computer Vision System

A computer vision system automatically recognizes facial expressions of students during problem solving.

Littlewort, Phan, Reilly, and Bartlett, UC San Diego and SDSU

There has been growing recognition of the importance of adaptive tutoring systems that respond to the student’s emotional and cognitive state. However little is known about children’s facial expressions during a problem solving task. What are the actual signals of boredom, interest, confusion, or uncertainty in real, spontaneous behavior of students? The field also is in need of spontaneous datasets to drive automated recognition of these states. TDLC researchers have collected a dataset of 50 children ages 3-9 during a set of problem solving tasks. Their behavior was measured. Figure 1 shows a set of facial movements automatically measured using the computer expression recognition toolbox (CERT). Shown is a time-warped average over 50 videos of 9 children in a haptic problem solving task. As the children solved thee problem, chin raise and corrugator (brow lower) movements decreased, while zygomatic (smile) increased. Individual facial responses were ballistic rather than linear. This research builds the foundation for automated tutoring systems that sense the state of the student and adapt accordingly.


facial expressions


Littlewort, Phan, Reilly and Bartlett (in prep). Automated measurement of children’s facial expressions in problem solving.