UC San Diego

Movements of the body beyond those involved in articulation are often exquisitely timed with speech. These embodied cues often inform semantic and pragmatic levels of social understanding but are also relevant for shaping how attention is organized to a speaker’s behavior and for the interpretation of prosodic speech features such as lexical stress. For these reasons, joint action between speech and gesture has been popularized by the term multimodal prosody, however, neurocognitive theories cannot yet account for the way that gestural cues interact with auditory speech processing or describe the necessary conditions for them to do so.As outlined in Chapter 1, the Sensorimotor Account of Multimodal Prosody (SAMP) forages from relatively disparate fields of inquiry to build a neurobiologically plausible hypothesis about mechanisms that service multisensory integration and attentional alignment during spoken language exchange. The SAMP hypothesizes that neural mechanisms tied to self-motion (vestibular) variables may be relevant neural codes that could be used to enforce the instantiation of temporally coupled communicative behaviors during discourse production, and that associated mechanisms assist in coding similar latent variables during discourse comprehension. Chapter 2 investigates a critical prediction of the SAMP: that kinetic, or force-related properties of co-speech movements are informative about a speaker’s vocal output. By training machine learning algorithms to predict amplitude characteristics of speech from whole-body motion capture data, we provide evidence that gravity-compatible movement trajectories are important for learning contingencies between and speech and co-speech movements. This validates the claim that kinetic properties of co-speech movement are informative for a speaker’s vocal output in ecologically relevant contexts. Chapters 3 and 4 use time-sensitive neuroimaging techniques (EEG) to investigate the potential influence of co-speech movement kinematics on auditory speech processing and provide novel evidence for multisensory enhancement effects of co-speech gesture dynamics on the neural representation of speech acoustics. Finally, Chapter 5 provides evidence that kinetic properties of co-speech gestures are relevant for driving multisensory interactivity during discourse processing, as the neural representation of kinetically informative biological motion trajectories were disproportionately impacted by the speaker’s voice. Compatible with radically embodied, biomechanical accounts of multimodal prosody, the SAMP provides a starting point for explaining how audiovisual interactions during discourse comprehension might be explained at this scale. This dissertation offers a theoretically grounded and empirical look into the way bodily movements traditionally defined as “nonverbal” may hold a more direct relationship with spoken language in the brain than is currently recognized. The presented work also motivates current neurocognitive models of speech processing to expand the scope of multisensory interactions that they currently account for–broadening an area of study that has historically focused on phonetics into the domain of prosody.