Assessing the effect of beat gestures on the perception of lexical stress in full Spanish utterances

Abstract

In everyday communication, interlocutors typically both hear and see each other. The auditory and visual modalities interact in speech perception, for example when it comes to the perception of lexical stress. A visual cue to lexical stress is provided by beat gestures – brief up-and-down movements of the hand. Indeed, a Spanish listener may perceive HAblo (“I speak”) or haBLÓ (“(s)he spoke”), depending on which syllable has a visual beat gesture aligned to it. This so-called “Manual McGurk effect” has primarily been demonstrated using isolated words. By embedding words in sentences, this online study tested the effect of beat gestures on the perception of lexical stress for more naturalistic utterances. The results from a 2-AFC task indicated that, in sentences, beat gestures did not reliably influence Spanish stress perception. Similarly, reaction time analyses revealed no reliable effect of beat gestures on the perception of lexical stress. The results may relate to differences in speech rate or variability in the gestures. However, these factors better represent naturalistic circumstances. The results are discussed from a cue integration perspective, where the beat gesture may be weighted relatively lightly in the process of Spanish lexical stress perception in full sentences.