The timing of an avatar’s beat gestures biases lexical stress perception in vocoded speech

Abstract

Cochlear implants (CIs) are neural prostheses that restore some level of hearing capacity, albeit conveying a less fine-grained speech signal than normal hearing conditions. For example, CIs convey altered fundamental frequency (F0) information, resulting in atypical lexical stress perception (e.g., distinguishing between the noun CONtent and the adjective conTENT) in languages in which this feature rests on F0 modulations. CI users can compensate for the degraded nature of the acoustic input by exploiting the audiovisual affordances of human communication, weighing more heavily the visual information provided by speakers (e.g., lip movements and gestures). Recent studies showed that, in individuals with normal hearing, the timing of simple up-and-down movements of the hand (i.e., beat gestures) biases lexical stress perception. The present study tested if the timing of beat gestures produced by an avatar can bias Dutch lexical stress perception in vocoded speech, which limits the reliability of F0 information in a way that mimics CI-hearing conditions. The effect of gestures in vocoded speech was particularly pronounced when hearing an ambiguous or the least frequent stress pattern in Dutch. These results suggest that (even artificially generated) beat gestures can support the perception of vocoded speech, especially when processing less frequent prosodic features.