Home Research Research Highlights
Hand Actions Guide Sentence Completion

The representation of hand actions in auditory sentence comprehension

Daniel Bub, University of Victoria

Recent evidence reveals that knowledge of actions may be crucial to understanding the meaning of sentences. Listeners may be simulating the actions implied by the sentence as part of the procedures that extract meaning. A recent study by Daniel Bub, affiliated with the NSF-funded Temporal Dynamics Learning Center, and his colleagues considers the role that these mental simulations may play in comprehension. The team developed a method to measure the activation of hand action representations time-locked to the onset of a word in a sentence. They asked whether, and at what point in time, a word like calculator activates the hand actions used to grasp it (volumetric actions) and the actions required to implement its function (functional actions). The team found that a word like calculator, presented in isolation (i.e. not in a sentence context) evokes only functional hand action representations and not volumetric hand actions. The same is true of sentences that do not describe any physical contact with the object (e.g. John looked at the calculator). But surprisingly, sentences describing physical interaction with objects, like “kick” and “step on” activate hand actions associated with grasping and lifting the object. The scientists believe that motor representations associated with holding and lifting an object become important when we need to compute the object’s shape and weight, and this information becomes necessary in comprehending sentences like “The angry lawyer kicked the calculator”. Mental simulations of motor actions can provide information that gives us the flexibility to interpret sentences that refer to unusual or unexpected physical interactions with objects. We may not know exactly what it feels like to kick a calculator, but we can rely on sensory-motor representations based on what it felt like in the past to lift and move such an object by hand. Thus, in certain situations, the simulation provides a kind of analogy for the listener, using whatever prior motor representations are available.

A multi-element response apparatus consisting of generic shapes associated through training with eight distinct hand actions. Subjects learned to produce distinct hand actions cued by a photograph of a hand posture, where each cue signaled a particular action carried out on the apparatus. Immediately before carrying out a cued hand action, subjects heard an auditory sentence referring to a manipulable object.
Masson, M. E. J., Bub, D. N., & Warren, C. M. (2008). Kicking calculators: Contribution of embodied representations to sentence comprehension. Journal of Memory and Language, 59, 256-265