Biology subjects

Suess, N.

Publications and source records attributed to Suess, N..

7 recordsLinked to original sources

bioRxiv2024

Eye Movements in Silent Visual Speech track Unheard Acoustic Signals and Relate to Hearing Experience

Behavioral and neuroscientific studies have shown that watching a speakers lip movements aids speech comprehension. Intriguingly, even when videos of speakers are presented silently, various cortical regions track auditory features, such as the envelope. Recently, we demonstrated that eye movements track low-level acoustic information when attentively listening to speech. In this study we investigated whether ocular speech tracking occurs during visual speech and how it influences cortical silent speech tracking. Furthermore, we compared the data of hearing individuals with congenitally deaf individuals, and those with acquired deafness or hearing loss (DHH; Deaf or hard of hearing) to assess how auditory deprivation (early vs. late onset) affects neural and ocular speech tracking during silent lip-reading. Using magnetoencephalography (MEG), we examined ocular and neural speech tracking of 75 participants observing silent videos of a speaker played forward and backward. Our main finding is a clear ocular unheard speech tracking effect with a dominance <1 Hz, which was not present for the lip movements. Similarly, we observed a <=1.3 Hz effect of neural unheard speech tracking in temporal regions for hearing participants. Importantly, neural tracking was not directly linked to ocular tracking in this study. Strikingly, across different listening groups, deaf participants with auditory listening experience showed higher ocular speech tracking than hearing participants, while no ocular speech tracking effect was revealed for congenitally deaf participants in a very small sample. This study extends our previous work by demonstrating the involvement of eye movements in speech processing, even in the absence of acoustic input.

neuroscience↗

bioRxiv2023

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker's mouth is occluded

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

neuroscience↗

bioRxiv2023

Individual prediction tendencies are related but not the same across modalities

Predictive processing theories, which model the brain as a "prediction machine", explain a wide range of cognitive functions, including learning, perception and action. Furthermore, it is increasingly accepted that aberrant prediction tendencies play a crucial role in psychiatric disorders. Given this explanatory value for clinical psychiatry, prediction tendencies are often implicitly conceptualised as individual traits or as tendencies that generalise across situations. As this has not yet explicitly been shown, in the current study, we quantify to what extent the individual tendency to anticipate sensory features of high probability generalises across modalities. Using magnetoencephalography (MEG), we recorded brain activity while participants were presented a sequence of four different (either visual or auditory) stimuli, which changed according to predefined transitional probabilities of two entropy levels: ordered vs. random. Our results show that, on a group-level, under conditions of low entropy, stimulus features of high probability are preactivated in the auditory but not in the visual modality. Crucially, the magnitude of the individual tendency to predict sensory events seems not to correlate between the two modalities. Furthermore, reliability statistics indicate poor internal consistency, suggesting that the measures from the different modalities are unlikely to reflect a single, common cognitive process. In sum, our findings suggest that quantification and interpretation of individual prediction tendencies cannot be generalised across modalities.

neuroscience↗

bioRxiv2021

Face masks impair reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Multisensory integration enables stimulus representation even when the sensory input in a single modality is weak. In the context of speech, when confronted with a degraded acoustic signal, congruent visual inputs promote comprehension. When this input is occluded speech comprehension consequently becomes more difficult. But it still remains inconclusive which levels of speech processing are affected under which circumstances by occlusion of the mouth area. To answer this question, we conducted an audiovisual (AV) multi-speaker experiment using naturalistic speech. In half of the trials, the target speaker wore a (surgical) face mask, while we measured the brain activity of normal hearing participants via magnetoencephalography (MEG). We additionally added a distractor speaker in half of the trials in order to create an ecologic difficult listening situation. A decoding model on the clear AV speech was trained and used to reconstruct crucial speech features in each condition. We found significant main effects of face masks on the reconstruction of acoustic features, such as the speech envelope and spectral speech features (i.e. pitch and formant frequencies), while reconstruction of higher level features of speech segmentation (phoneme and word onsets) were especially impaired through masks in difficult listening situations. As we used surgical face masks in our study, which only show mild effects on speech acoustics, we interpret our findings as the result of the occluded lip movements. This idea is in line with recent research showing that visual cortical regions track spectral modulations. Our findings extend previous behavioural results, by demonstrating the complex contextual effects of occluding relevant visual information on speech processing. HighlightsO_LISurgical face masks impair neural tracking of speech features C_LIO_LITracking of acoustic features is generally impaired, while higher level segmentational features show their effects especially in challenging listening situations C_LIO_LIAn explanation is the prevention of a visuo-phonological transformation contributing to audiovisual multisensory integration C_LI

neuroscience↗

bioRxiv2021

Age-related decreases of cortical visuo-phonological transformation of unheard spectral fine-details

The integration of visual and auditory cues is crucial for successful processing of speech, especially under adverse conditions. Recent reports have shown that when participants watch muted videos of speakers, the phonological information about the acoustic speech envelope is tracked by the visual cortex. However, the speech signal also carries much richer acoustic details, e.g. about the fundamental frequency and the resonant frequencies, whose visuo-phonological transformation could aid speech processing. Here, we investigated the neural basis of the visuo-phonological transformation processes of these more fine-grained acoustic details and assessed how they change with ageing. We recorded whole-head magnetoencephalography (MEG) data while participants watched silent intelligible and unintelligible videos of a speaker. We found that the visual cortex is able to track the unheard intelligible modulations of resonant frequencies and the pitch linked to lip movements. Importantly, only the processing of intelligible unheard formants decreases significantly with age in the visual and also in the cingulate cortex. This is not the case for the processing of the unheard speech envelope, the fundamental frequency or the purely visual information carried by lip movements. These results show that unheard spectral fine-details (along with the unheard acoustic envelope) are transformed from a mere visual to a phonological representation. Aging affects especially the ability to derive spectral dynamics at formant frequencies. Since listening in noisy environments should capitalize on the ability to track spectral fine-details, our results provide a novel focus on compensatory processes in such challenging situations.

neuroscience↗

bioRxiv2019

Phantom auditory perception (tinnitus) is characterised by stronger anticipatory auditory predictions

How phantom perceptions arise and the factors that make individuals prone to such experiences are not well understood. An attractive phenomenon to study these questions is tinnitus, a very common auditory phantom perception which is not explained by hyperactivity in the auditory pathway alone. Our framework posits that a predisposition to developing (chronic) tinnitus is dependent on individual traits relating to the formation and utilization of sensory predictions. Predictions of auditory stimulus frequency (remote from tinnitus frequency) were studied using a paradigm parametrically modulating regularity (i.e. predictability) of tone sequences and applying decoding techniques on magnetoencephalographic (MEG) data. For processes likely linked to short-term memory, individuals with tinnitus showed an enhanced anticipatory prediction pattern associated with increasing sequence regularity. In contrast, individuals without tinnitus engaged the same processes following the onset of the to-be-decoded sound. We posit that this tendency to optimally anticipate static and changing auditory inputs may determine which individuals faced with persistent auditory pathway hyperactivity factor it into auditory predictions, and thus perceive it as tinnitus. While our study constitutes a first step relating vulnerability to tinnitus with predictive processing, longitudinal studies are needed to confirm the predisposition model of tinnitus development.

neuroscience↗

bioRxiv2019

Supramodal selective attention differentially adjusts frequency and phase of entrained oscillations in primary sensory areas and the dorsal attention network

Continuously prioritizing behaviourally relevant information from the environment for improved stimulus processing is a crucial function of attention. Low-frequency phase alignment of neural activity in primary sensory areas, with respect to attended/ignored features has been suggested to support top-down prioritization. Phase adjustment in frontoparietal regions has not been widely studied, despite general implication of these in top-down selection of information. In the current MEG study, we investigated how ongoing oscillatory activity of both sensory and non-sensory brain regions are differentially impacted by attentional focus. Participants performed an established intermodal selective attention task, where low-frequency auditory (1.6 Hz) and visual (1.8 Hz) stimuli were presented simultaneously. We instructed participants to either attend to the auditory or to the visual stimuli and to detect targets while ignoring the other stimulus stream. As expected, the strongest phase adjustment was observed in primary sensory regions for auditory and for visual stimulation, independent of attentional focus. We found greater differences in phase locking between attended and ignored stimulation for the visual modality. Interestingly, auditory temporal regions show small but significant attention-dependent entrainment even for visual stimulation. Extending findings from invasive recordings in non-human primates, we demonstrate an effect of attentional focus on the phase of the entrained oscillations in auditory and visual cortex which may be driven by phase locked increases of induced power. In contrast to the effects in sensory areas, attentional focus adjusted the peak frequencies in nonsensory areas. Spatially these areas show a striking overlap with core regions of the dorsal attention network and the frontoparietal network. This suggests that these areas prioritize the attended modality by optimally exploiting the temporal structure of stimulation. Overall, our study complements and extends previous work by showing a differential effect of attentional focus on entrained oscillations in primary sensory areas and frontoparietal areas.

neuroscience↗