Beat gestures made by human-like avatars affect speech perception

Abstract

In face-to-face communication, several visual cues support speech perception. Even the timing of simple up-and-down flicks of the hand, called beat gestures, can convey word stress, changing what individuals hear (e.g., CONtent vs. conTENT). While beat gestures have been traditionally investigated in human-to-human communications, nowadays individuals increasingly interact with computer-controlled avatars (e.g., virtual assistants). The present study tested whether beat gestures produced by an avatar affect word stress perception, similarly to human gestures. Furthermore, this study tested whether a minimal visual cue such as a 2D moving disc can also affect speech perception. Beat gestures made by the avatar significantly affected speech perception, albeit slightly less than human-made gestures. The disc condition did not affect speech perception. The present work lays the foundation for the application of (beat) gesturing avatars, which could be used to boost speech intelligibility.