Human-Robot Interaction (HRI) as a Tool to Monitor Socio-Emotional Development in Early Childhood Education


TDLC’s Moshen Malmir, Deborah Forster and Javier Movellan in the Machine Perception Laboratory, in collaboration with Kathryn Owens and Lydia Morrison from UC San Diego’s Early Childhood Education Center (ECEC), demonstrated (Malmir et al., 2013; Movellan et al., 2014) the potential of using social educational robots in the classroom, not only successfully matching staff’s independent evaluation of children’s game preference, by successfully capturing affective behavior (via facial expression recognition,) but also monitoring relational aspects of spontaneous behavior among young children (see Figure 1).

Adding to an earlier study showing that presence of the robot can improve vocabulary acquisition, the study illustrates that social robots may become a useful tool in early childhood education to discover socio-emotional patterns over time and to monitor their development. The data provided by the robots could be used by educators and clinicians to discover problems and provide appropriate interventions.

The RUBI project started in 2004 with the goal of studying the potential value of social robot technologies in early childhood education environments. Previously, RUBI has been used in classrooms for educational purposes. In a 2-week field study, for example, we showed that RUBI-4 could teach kids English and Finnish vocabulary. The data for that study consisted of the game-log recorded automatically by RUBI, and continuous video and audio capture from an array of webcams and microphones. The video data had to be analyzed manually, a prohibitively labor-intensive effort that took well over a year to complete. Such a lengthy process restricts the usefulness of the data as responsive monitoring and/or intervention. To overcome such restrictions, automatic person recognition capabilities were added to RUBI-5 (Malmir et al., 2013; Movellan et al., 2014) so that all data streams (activity logs, facial expression recognition, and person recognition) were captured and processed automatically. We showed how the data collected by RUBI-5 could be used to predict kids preference over different activities using facial expression recognition, while keeping track of the co-presence of kids in RUBI’s ‘view’ - to produce the RUBIgram in Figure1 (after subtracting the associations that could have happened just by chance). The long-term collaboration with the staff, parents and young students of ECEC provide a rich environment in which educational and research expertise are brought to bear on a continually evolving platform in service of contemporary challenges and taking advantage of cutting-edge technological and computational developments. We call this immersive iterative design.