Biology subjects

Hauswald, A.

Publications and source records attributed to Hauswald, A..

13 recordsLinked to original sources

bioRxiv2025

Ocular speech tracking persists in blindness, but its dynamics and oculo-cerebral connectivity depend on visual status

While eye movements have been shown to track the speech envelope, it is unknown whether this reflects a hard-wired mechanism or one shaped by (lifetime) audiovisual experience. Further, questions remain about whether ocular tracking is modulated by speech intelligibility and which brain regions drive these synchronized eye movements. Here, we investigate ocular speech tracking in blindfolded early blind, late blind, and sighted individuals using magnetoencephalography (MEG) and source-reconstructed oculomotor signals while participants listened to narrative speech of varying intelligibility. We found that oculomotor activity tracks acoustic speech features and, unlike neural speech tracking, is not modulated by intelligibility. Interestingly, we found effects reflected in two frequency-specific components: a low-frequency ([~]1 Hz) effect present across all groups, indicating that visual experience is not required, and a high-frequency ([~]6 Hz) effect reduced in early- and late-blind individuals. Moreover, this finding is not driven by cerebro-ocular connectivity, as late-blind individuals exhibit stronger connectivity between the eyes and the left temporal cortices without a corresponding increase in ocular tracking. In conclusion, ocular speech tracking seems to respond selectively to acoustic but not to intelligibility features of speech, and it does not require visual experience to develop. It may thus represent a hard-wired oculomotor mechanism within the oculo-cerebral network involved in speech processing. Significance StatementEye movements provide a unique window into the interaction between auditory and visual systems. By studying early blind, late blind, and sighted individuals, we demonstrate that speech-related eye movements arise from at least two distinct mechanisms: a low-frequency component that occurs independently of (lifetime) visual experience and is linked to processing of acoustic speech features, and a high-frequency component shaped by prior visual exposure. Importantly, speech intelligibility - unlike its impact on neural measures - does not modulate these ocular responses. This dissociation suggests that eye movements reflect mechanisms of spoken language processing that are independent of intelligibility, thereby revealing novel pathways of auditory-motor coupling and broadening our understanding of sensory integration in the absence of vision.

neuroscience↗

bioRxiv2024

Eye Movements in Silent Visual Speech track Unheard Acoustic Signals and Relate to Hearing Experience

Behavioral and neuroscientific studies have shown that watching a speakers lip movements aids speech comprehension. Intriguingly, even when videos of speakers are presented silently, various cortical regions track auditory features, such as the envelope. Recently, we demonstrated that eye movements track low-level acoustic information when attentively listening to speech. In this study we investigated whether ocular speech tracking occurs during visual speech and how it influences cortical silent speech tracking. Furthermore, we compared the data of hearing individuals with congenitally deaf individuals, and those with acquired deafness or hearing loss (DHH; Deaf or hard of hearing) to assess how auditory deprivation (early vs. late onset) affects neural and ocular speech tracking during silent lip-reading. Using magnetoencephalography (MEG), we examined ocular and neural speech tracking of 75 participants observing silent videos of a speaker played forward and backward. Our main finding is a clear ocular unheard speech tracking effect with a dominance <1 Hz, which was not present for the lip movements. Similarly, we observed a <=1.3 Hz effect of neural unheard speech tracking in temporal regions for hearing participants. Importantly, neural tracking was not directly linked to ocular tracking in this study. Strikingly, across different listening groups, deaf participants with auditory listening experience showed higher ocular speech tracking than hearing participants, while no ocular speech tracking effect was revealed for congenitally deaf participants in a very small sample. This study extends our previous work by demonstrating the involvement of eye movements in speech processing, even in the absence of acoustic input.

neuroscience↗

bioRxiv2024

Decreasing hearing ability does not lead to improved visual speech extraction as revealed in a neural speech tracking paradigm

The use of visual speech is thought to be especially important in situations where acoustics are unclear and in individuals with hearing impairment. To investigate this in a neural speech tracking paradigm, we measured MEG in sixty-seven mid- to old-age individuals during audiovisual (AV), audio-only (A), and visual-only (V) speech in the context of face masks. First, we could extend previous findings by showing that not only in young normal-hearing individuals but also in aging individuals with decreasing hearing ability the brain is superior in neurally tracking the acoustic spectrogram in AV compared to A presentations, especially in multi-speaker situations. The addition of visual lip movements further increases this benefit. Second, we could show that neural speech tracking in individuals with lower levels of hearing ability is affected more by face masks. However, in this population, the effect seems to be a composite of blocked visual speech and distorted acoustics. Third, we could confirm previous findings, that the neural benefit of visual speech varies strongly across individuals. We show that this general individual ability predicts how much people engage in visual speech tracking in difficult AV listening situations. Interestingly, this was not correlated with hearing thresholds and therefore does seem to be a widely used compensatory strategy in the hearing impaired.

neuroscience↗

bioRxiv2024

Carrier-frequency specific omission-related neural activity in ordered sound sequences are independent of omissions-predictability

Regularities in our surroundings lead to predictions about upcoming events. Previous research has shown, that omitted sounds during otherwise regular tone sequences elicit frequency-specific neural activity related to the upcoming but omitted tone. We tested whether this neural response is depending on the unpredictability of the omission. Therefore we recorded MEG data while participants listened to ordered or random tone sequences with omissions occurring either ordered or randomly. Using multivariate pattern analysis shows that the frequency-specific neural pattern during omission within ordered tone sequences occurs independent of the regularity of the omissions suggesting that the auditory predictions based on sensory experiences are not immediately updated by violations of those expectations.

neuroscience↗

bioRxiv2023

Lip movements and lexical features improve speech tracking differently for clear and multi-speaker speech

Visual speech plays a powerful role in facilitating auditory speech processing and has been a publicly noticed topic with the wide usage of face masks during the Covid-19 pandemic. In a previous magnetoencephalography (MEG) study we showed that occluding the mouth area significantly impairs neural speech tracking. To rule out the possibility that this deterioration is due to degraded sound quality, in the present follow-up study, we presented participants with audiovisual (AV) and audio-only (A) speech. We further independently manipulated the trials by adding a face mask and a distractor speaker. Our results clearly show that face masks only affect speech tracking in AV conditions, not in A conditions. This shows that face masks indeed primarily impact speech processing by blocking visual speech and not by acoustic degradation. Furthermore, we observe differences in the speech features that are used for visual speech processing. On the one hand, processing in clear speech, but not in noisy speech, is profiting more from lexical unit features (phonemes and word onsets) hinting at improved phoneme discrimination. On the other hand, we observe an improvement in speech tracking driven by the modulations of the lip area in clear speech and conditions with a distractor speaker, which might aid by providing temporal cues for subsequent auditory processing. With this work, we highlight the effects of face masks in AV speech tracking and show two separate ways how visual speech might support successful speech processing.

neuroscience↗

bioRxiv2023

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker's mouth is occluded

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

neuroscience↗

bioRxiv2022

Speech intelligibility changes the temporal evolution of neural speech tracking

Listening to speech with poor signal quality is challenging. Neural speech tracking of degraded speech has been used to advance the understanding of how brain processes and speech intelligibility are interrelated, however the temporal dynamics of neural speech tracking are not clear. In the present MEG study, we thereby exploited temporal response functions (TRFs) and generated signal-degraded speech to depict the temporal evolution of speech intelligibility modulation on neural speech tracking. In addition, we inter-related facets of neural speech tracking (e.g., speech envelope reconstruction, speech-brain coherence, and components of broadband coherence spectra) to endorse our findings in TRFs. Our TRF analysis yielded marked temporally differential effects of vocoding: reduction of intelligibility went along with large increases of early peak responses ([~]50-110 ms, M50TRF), but strongly reduced responses around 175-230 ms (M200TRF). For the late responses 315-380 ms (M350TRF), the maximum response occurred for degraded speech that was still comprehensible then declined with reduced intelligibility. Furthermore, we related the TRF components to our other neural "tracking" measures and found that M50TRF and M200TRF play a differential role in the shifting center frequency of the broadband coherence spectra. Overall, our study highlights the importance of time-resolved computation and parametrization of coherence spectra on neural speech tracking and provides a better understanding of degraded speech processing. HighlightsO_LIWe use MEG to show that speech intelligibility differentially impacts the temporal evolution of neural speech tracking. C_LIO_LITRF responses around 200 ms show the strongest relationship with behaviour. C_LIO_LIRelating TRF effects to parameterized coherence spectra using FOOOF suggests that M50TRF and M200TRF reflect shifts in which speech features are tracked over time. C_LI

neuroscience↗

bioRxiv2021

Face masks impair reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Multisensory integration enables stimulus representation even when the sensory input in a single modality is weak. In the context of speech, when confronted with a degraded acoustic signal, congruent visual inputs promote comprehension. When this input is occluded speech comprehension consequently becomes more difficult. But it still remains inconclusive which levels of speech processing are affected under which circumstances by occlusion of the mouth area. To answer this question, we conducted an audiovisual (AV) multi-speaker experiment using naturalistic speech. In half of the trials, the target speaker wore a (surgical) face mask, while we measured the brain activity of normal hearing participants via magnetoencephalography (MEG). We additionally added a distractor speaker in half of the trials in order to create an ecologic difficult listening situation. A decoding model on the clear AV speech was trained and used to reconstruct crucial speech features in each condition. We found significant main effects of face masks on the reconstruction of acoustic features, such as the speech envelope and spectral speech features (i.e. pitch and formant frequencies), while reconstruction of higher level features of speech segmentation (phoneme and word onsets) were especially impaired through masks in difficult listening situations. As we used surgical face masks in our study, which only show mild effects on speech acoustics, we interpret our findings as the result of the occluded lip movements. This idea is in line with recent research showing that visual cortical regions track spectral modulations. Our findings extend previous behavioural results, by demonstrating the complex contextual effects of occluding relevant visual information on speech processing. HighlightsO_LISurgical face masks impair neural tracking of speech features C_LIO_LITracking of acoustic features is generally impaired, while higher level segmentational features show their effects especially in challenging listening situations C_LIO_LIAn explanation is the prevention of a visuo-phonological transformation contributing to audiovisual multisensory integration C_LI

neuroscience↗

bioRxiv2021

Age-related decreases of cortical visuo-phonological transformation of unheard spectral fine-details

The integration of visual and auditory cues is crucial for successful processing of speech, especially under adverse conditions. Recent reports have shown that when participants watch muted videos of speakers, the phonological information about the acoustic speech envelope is tracked by the visual cortex. However, the speech signal also carries much richer acoustic details, e.g. about the fundamental frequency and the resonant frequencies, whose visuo-phonological transformation could aid speech processing. Here, we investigated the neural basis of the visuo-phonological transformation processes of these more fine-grained acoustic details and assessed how they change with ageing. We recorded whole-head magnetoencephalography (MEG) data while participants watched silent intelligible and unintelligible videos of a speaker. We found that the visual cortex is able to track the unheard intelligible modulations of resonant frequencies and the pitch linked to lip movements. Importantly, only the processing of intelligible unheard formants decreases significantly with age in the visual and also in the cingulate cortex. This is not the case for the processing of the unheard speech envelope, the fundamental frequency or the purely visual information carried by lip movements. These results show that unheard spectral fine-details (along with the unheard acoustic envelope) are transformed from a mere visual to a phonological representation. Aging affects especially the ability to derive spectral dynamics at formant frequencies. Since listening in noisy environments should capitalize on the ability to track spectral fine-details, our results provide a novel focus on compensatory processes in such challenging situations.

neuroscience↗

bioRxiv2021

Neural speech tracking shifts from the syllabic to the modulation rate of speech as intelligibility decreases

The most prominent acoustic features in speech are intensity modulations, represented by the amplitude envelope of speech. Synchronization of neural activity with these modulations is vital for speech comprehension. As the acoustic modulation of speech is related to the production of syllables, investigations of neural speech tracking rarely distinguish between lower-level acoustic (envelope modulation) and higher-level linguistic (syllable rate) information. Here we manipulated speech intelligibility using noise-vocoded speech and investigated the spectral dynamics of neural speech processing, across two studies at cortical and subcortical levels of the auditory hierarchy, using magnetoencephalography. Overall, cortical regions mostly track the syllable rate, whereas subcortical regions track the acoustic envelope. Furthermore, with less intelligible speech, tracking of the modulation rate becomes more dominant. Our study highlights the importance of distinguishing between envelope modulation and syllable rate and provides novel possibilities to better understand differences between auditory processing and speech/language processing disorders. O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=160 SRC="FIGDIR/small/437033v1_ufig1.gif" ALT="Figure 1"> View larger version (23K): org.highwire.dtl.DTLVardef@6364adorg.highwire.dtl.DTLVardef@166daaeorg.highwire.dtl.DTLVardef@19eedd2org.highwire.dtl.DTLVardef@10f7aed_HPS_FORMAT_FIGEXP M_FIG C_FIG

neuroscience↗

bioRxiv2019

Degraded auditory and visual speech affects theta synchronization and alpha power differently

Understanding degraded speech, e.g. following a hearing damage, can pose a challenge. Previous attempts to quantify speech intelligibility in neural terms have usually focused on one of two measures, namely low-frequency speech-brain synchronization or alpha power modulations. However, reports have been mixed concerning the modulation of these measures, an issue aggravated by the fact that they have normally been studied separately. Using a parametric speech degradation approach, we present MEG studies that overcome this shortcoming. In a first study, participants listened to unimodal auditory speech with three different levels of degradation (original, 7-channel and 3-channel vocoding). Intelligibility declined with declining clarity, implemented by fewer vocoding channels but was still intelligible to some extent even for the lowest clarity level used (3-channel vocoding). Low- frequency (1-7 Hz) speech tracking suggested a u-shaped relationship with strongest effects for the medium degraded speech (7-channel) in bilateral auditory and left frontal regions. To follow up on this finding, we implemented three additional vocoding levels (5-channel, 2- channel, 1-channel) in a second MEG study. Using this wider range of degradation, the speech-brain synchronization showed a similar pattern for the degradation levels used in the first study but further shows that when speech becomes unintelligible, synchronization declines again. The relationship differed for alpha power, which continued to decrease across vocoding levels reaching a floor effect for 5-channel vocoding. Predicting subjective intelligibility based on models either combining both measures or each measure alone, showed superiority of the combined model. Our findings underline that speech tracking and alpha power are modified differently by the degree of degradation of continuous speech but together contribute to the subjective understanding of speech.

neuroscience↗

bioRxiv2019

Head MagnetoMyography (hMMG): a novel approach to monitor face and whole-head muscular activity

Muscular activity recording is of high basic science and clinical relevance and is typically achieved using electromyography (EMG). While providing detailed information about the state of a specific muscle, this technique has limitations such as the need for a-priori assumptions about electrode placement and difficulty with recording muscular activity patterns from extended body areas at once. For head and face muscle activity, the present work aimed to overcome these restrictions by exploiting magnetoencephalography (MEG) as a whole-head myographic recorder (head magnetomyography, hMMG). This is in contrast to common MEG studies, which treat muscular activity as artifact in electromagnetic brain activity. In a first proof-of-concept step, participants imitated emotional facial expressions performed by a model. Exploiting source projection algorithms, we were able to reconstruct muscular activity, showing spatial activation patterns in accord with the hypothesized muscular contractions. Going one step further, participants passively observed affective pictures with negative, neutral, or positive valence. Applying multivariate pattern analysis to the reconstructed hMMG signal, we were able to decode above chance the valence category of the presented pictures. Underlining the potential of hMMG, a searchlight analysis revealed that generally neglected neck muscles exhibit information on stimulus valence. Results confirm the utility of hMMG as a whole-head electromyographic recorder to quantify muscular activation patterns including muscular regions that are typically not recorded with EMG. This key advantage beyond conventional EMG has substantial scientific and clinical potential.

neuroscience↗

bioRxiv2018

Visual cortical entrainment to unheard acoustic speech reflects intelligibility of lip movements and is mediated by dorsal stream regions

Successful lip reading requires a mapping from visual to phonological information [1]. Recently, visual and motor cortices have been implicated in tracking lip movements (e.g. [2]). It remains unclear, however, whether visuo-phonological mapping occurs already at the level of the visual cortex, that is, whether this structure tracks the acoustic signal in a functionally relevant manner. In order to elucidate this, we investigated how the cortex tracks (i.e. entrains) absent acoustic speech signals carried by silent lip movements. Crucially, we contrasted the entrainment to unheard forward (intelligible) and backward (unintelligible) acoustic speech. We observed that the visual cortex exhibited stronger entrainment to the unheard forward acoustic speech envelope compared to the unheard backward acoustic speech envelope. Supporting the notion of a visuo-phonological mapping process, this forward-backward difference of occipital entrainment was not present for actually observed lip movements. Importantly, the respective occipital region received more top-down input especially from left premotor, primary motor, somatosensory regions and, to a lesser extent, also from posterior temporal cortex. Strikingly, across participants, the extent of top-down modulation of visual cortex stemming from these regions partially correlates with the strength of entrainment to absent acoustic forward speech envelope but not to present forward lip movements. Our findings demonstrate that a distributed cortical network, including key dorsal stream auditory regions [3-5], influence how the visual cortex shows sensitivity to the intelligibility of speech while tracking silent lip movements.\n\nHighlightsO_LIVisual cortex tracks better forward than backward unheard acoustic speech envelope\nC_LIO_LIEffects not \"trivially\" caused by correlation of visual with acoustic signal\nC_LIO_LIStronger top-down control of visual cortex during forward display of lip movements\nC_LIO_LITop-down influence correlates with visual cortical entrainment effect\nC_LIO_LIResults seem to reflect visuo-phonological mapping processes\nC_LI

neuroscience↗