Oscillatory and gaze signatures of socio-emotional speech processing, visuo-spatial cognition, and their interaction in a near-realistic dual-task MEG study

In everyday settings, information processing depends on attentional focus, available cognitive resources, and stimulus characteristics such as its valence. Despite insights from laboratory studies, multisensory information processing in complex, realistic environments remains poorly understood. Using whole-head magnetoencephalography and eye tracking in a driving simulation, we addressed two research questions: (1) What are the distinct signatures of (socio-)emotional speech processing and visuo-spatial cognition in the dual task of driving and listening and (2) is information processing co-modulated by valence of speech and workload? We hypothesised two antagonistic processing modes: A top-down guided mode for (socio-)emotional speech processing and a bottom-up stimulus-driven mode for visuo-spatial cognition under high workload. A third mode was proposed to regulate emotional interference during driving under high visuo-spatial load and limited cognitive capacity. Its involvement is hypothesised to be co-modulated by valence and workload. Spatial clustering of oscillatory source activity supported the hypotheses: During emotional compared with neutral speech, parietal beta-band power increased, which likely supported processes related to predictive listening and socio-emotional cognition. In this top-down guided mode, driving-related activity decreased, as indicated by greater parietal alpha- and reduced gamma-band power. During high visuo-spatial workload, linked to the bottom-up stimulus-driven mode, gamma-band power increased in motor and orbitofrontal cortices, whereas beta-band power decreased in motor and temporo-parietal regions. Moreover, increased pupil dilation and decreased gaze dispersion were associated with this bottom-up stimulus-driven mode and visuo-spatial cognitive demands. Multivariate pattern analyses identified the third regulatory mode reflected in fronto-temporal gamma-band oscillations. It was co-modulated by valence and workload: Under low workload drives, gamma-band activity increased for negative compared with positive speech, pointing to the recruitment of inhibitory control processes. Under high workload drives with heightened visuo-spatial demands, gamma-band activity increased for positive speech, but decreased for negative speech, suggesting reduced cognitive resources and impaired control. To conclude, we found converging evidence of distinct signatures for (socio-)emotional speech, visuo-spatial cognition, and interference regulation in naturalistic multisensory environments. We propose that the top-down guided mode represents anticipatory listening and socio-emotional cognition, whereas the bottom-up stimulus-driven mode reflects the allocation of resources to driving-related spatial and sensorimotor processing but also cognitive strain under heightened task demands. Fronto-temporal gamma oscillations likely enable flexible up- and down-regulation of emotional speech processing in response to potential interference in complex, naturalistic environments.

Imaging Neuroscience