Paper published in Applied Psycholinguistics!
Movements made by an avatar can support listening to vocoded speech. Read all about it in our new publication in Applied Psycholinguistics
Published!
Our new paper “The timing of an avatar’s beat gestures biases lexical stress perception in vocoded speech”, written by Matteo Maran, Renske M. J. Uilenreef, Roos Rossen, and Hans Rutger Bosker was published in Applied Psycholinguistics. The full text and all supporting data are publicly available from links at the bottom of this post.
What’s it about?
Cochlear implants (CIs) are neural prostheses that can restore some level of hearing capacity. However, CIs convey a signal that distorts some aspects of speech, for example the intonation (“fundamental frequency”) that provides the emphasis to distinguish between two otherwise identical words (e.g., the noun “CONtent” vs. the adjective “conTENT”, which differ in the so-called lexical stress). Recent studies show that simple up-and-down flick of the hands, called beat gestures, affect lexical stress: a beat gesture falling on the first syllable of a word (e.g., “content”) will make it more prominent, making listeners more likely to hear the word with emphasis on the first syllable (e.g., hearing the noun “CONtent”, rather than the adjective “conTENT”). Similarly, a beat gesture falling on the second syllable will provide it with more emphasis (e.g., hearing the adjective “conTENT”, rather than the noun “CONtent”).
In this online experiment, we tested whether beat gestures can affect the perceived emphasis of words once access to the fundamental frequency information is limited (“vocoded speech”), in a way that might resemble hearing through CIs. Clear speech served as a baseline condition for comparison. Additionally, the beat gestures were made by a human-looking avatar, which could in principle be used in future app-based software to support hearing through CIs. Beat gestures indeed affected the perceived emphasis of words in vocoded speech, especially when the information in the speech signal was ambiguous (e.g., half-way between CONtent and conTENT) or infrequent (e.g., emphasis on the second syllable, which in the tested language occurs only 12% of the time). In these cases, the effect of beat gestures in vocoded speech was numerically larger than in clear speech.
Why is this important?
This research shows that beat gestures can affect which words individuals hear once the speech signal is distorted. This property of beat gestures might be particularly important for individuals who are routinely exposed to distorted speech, for example because they hear through cochlear implants. Importantly, our study shows that even beat gestures made by a human-looking avatar affect speech perception. Therefore, the present findings might inspire the development of apps and software in which avatars, by moving to the rhythm of speech, support comprehension in individuals with altered hearing.
Full reference
The full citation, open access PDF, and all data are publicly available from the links below:
(2025). The timing of an avatar’s beat gestures biases lexical stress perception in vocoded speech. Applied Psycholinguistics 46, e37, doi:10.1017/S0142716425100180.