Ye et al. (2025) published in Attention, Perception, & Psychophysics!
The same gesture can be perceived differently, depending on the speech you hear!
Published!
Our new paper “Effect of auditory cues to lexical stress on the visual perception of gestural timing”, authored by Chengjia Ye, James McQueen and Hans Rutger Bosker, is now publicly available online! It’s published Open Access in the journal Attention, Perception, & Psychophysics. Feel free to check it out: links are given at the bottom of the page.
What’s it about?
We ran two experiments to see how native listeners of Dutch use auditory cues to lexical stress when perceiving the timing of hand beats. Previous studies of our lab have demonstrated that the timing of hand beats can influence the perception of lexical stress, and thus spoken-word recognition. In this study, we examined ‘a reverse effect’: does lexical stress change how you perceive the visual timing of hand gestures? Our results showed that the perceived timing of a hand beat is indeed ‘pulled towards’ the syllable with stronger stress cues. For example, our participants perceived an ’early gesture’, objectively preceding a stressed syllable by 200 ms, to only precede the stress by 80 ms! In fact, the same gesture, falling midway between two syllables, can be perceived as falling on the first or second syllable, depending on which of two syllables is acoustically stressed.
Why is this important?
In everyday conversations, people often gesture while they speak. Small rhythmic gestures, in particular, are timed in synchrony with our speech, mostly falling on stressed syllables. However, this is no strict rule of course: people can gesture in different ways and at different times. Our study demonstrates that our brain can handle some variability in the timing of hand gestures and word stress. That is, our brain perceptually shrinks this variability: gestures that are objectively timed ‘out-of-sync’ with speech are perceived as less ‘out-of-sync’ because the acoustic stress ‘pulls on’ the gesture. This means that even if you produce slightly out-of-sync gestures, your audience can still correctly interpret which words and syllables your gestures are emphasizing! Clever ay?
Full reference
The full citation, open access PDF, and all data are publicly available from the links below:
(2025). Effect of auditory cues to lexical stress on the visual perception of gestural timing. Attention, Perception, & Psychophysics, doi:10.3758/s13414-025-03072-z.