Biology subjects

Francart, T.

Publications and source records attributed to Francart, T..

At least 19 recordsLinked to original sources

bioRxiv2026

Interpersonal Synchronization of Brain and Body Tracks Attention and Listening Engagement

Purpose: Speech perception is a fundamental part of everyday communication that relies on more than simple identification of words and sentences. Attention and listening engagement both contribute to speech perception, while representing distinct aspects of the listening experience. Attention is typically associated with cognitive focus, whereas listening engagement additionally involves cognitive and affective immersion in sound. Despite their importance, these states remain difficult to disentangle, behaviorally and physiologically. Both have been linked to interpersonal synchronization (the synchronization of biobehavioral signals across individuals), raising questions about what this synchronization actually reflects. Method: In this study, we disentangled attention and listening engagement by independently manipulating both factors within a single experiment. Thirty participants listened to two simultaneously presented streams of meaningful speech and were instructed to focus on only one. Both attended and unattended stimuli were designed to be either engaging or non-engaging. Neural activity was recorded using EEG, while physiological responses were measured using heart rate and electrodermal activity. Results: Interpersonal synchronization was computed from neural and bodily signals, alongside a self-report measure of listening engagement and auditory attention decoding (AAD), a neural measure of selective attention. Interpersonal synchronization of all three modalities significantly predicted listening engagement, whereas neural interpersonal synchronization was the only measure that significantly predicted attention. These findings suggest that attention is primarily driven by cognitive processes represented in the brain, while listening engagement additionally involves affective processes that are more strongly reflected in bodily responses. Conclusions: Overall, this study demonstrates that different forms of interpersonal synchronization reflect distinct dimensions of the listening experience and supports interpersonal synchronization as a potential objective marker of listening engagement.

neuroscience↗

bioRxiv2026

Neural Tracking of Speech Envelope as an Index of Spatial Release from Masking

Understanding speech in noisy environments relies strongly on binaural cues such as interaural time and level differences (ITDs and ILDs), which support spatial hearing and the segregation of competing sound sources. When these cues are degraded, listeners experience substantial difficulty in complex acoustic environments. Behavioural measures of binaural benefit, such as binaural masking level differences, binaural intelligibility level differences, and spatial release from masking (SRM), are well established in normal-hearing (NH) listeners, but they require an active behavioural response. Neural speech tracking using electroencephalography (EEG) has emerged as a promising approach for quantifying neural processing of continuous speech, yet its sensitivity to spatial hearing cues remains insufficiently characterised. We investigated the neural correlates of spatial release from masking in NH listeners using EEG-based neural speech tracking. Nineteen participants listened to continuous Dutch speech stories presented with masking noise under two spatial configurations, collocated (S0N0) and spatially separated (S0N90), across multiple signal-to-noise ratios (SNRs). Neural tracking of the speech envelope was quantified using both envelope reconstruction and temporal response function (TRF) analyses. Spatial separation enhanced neural tracking of the target speech envelope, particularly at challenging SNRs where behavioural SRM was also observed. TRF analysis further revealed increased amplitudes and decreased latencies of late cortical components consistent with spatial unmasking effects. These findings demonstrate that neural speech tracking captures cortical signatures of spatial unmasking and closely reflects behavioural improvements in speech understanding. Establishing these relationships in NH listeners supports the development of objective neural measures for evaluating binaural benefit in difficult-to-test populations.

neuroscience↗

bioRxiv2026

When feedback backfires: investigating neurofeedback effects in a closed-loop auditory attention decoding paradigm

Selective auditory attention decoding (AAD) enables tracking which of multiple concurrent speakers a listener attends to and is a key building block for neuro-steered hearing devices. While AAD integrated in a closed-loop system with real-time neurofeedback (NFB) is hypothesized to improve decoding through neural adaptation and error-correction behaviour, the short-term behavioral and algorithmic impact of such a bilateral human-machine interaction remains poorly understood. Here we evaluated the effects of NFB on AAD accuracy and user experience in a single-session AAD paradigm with online NFB involving nineteen participants. They performed a selective listening task with enforced attention switches across four conditions: open-loop (OL), closed-loop with auditory gain feedback (CLA), closed-loop with visual feedback (CLV), and a condition with pseudo-auditory gain control (psCLA) decoupled from the participants individual neural activity. AAD was performed online using both subject-specific and subject-independent linear decoders on 5 s sliding windows, followed by Hidden Markov Model post-processing. Online analysis showed comparable decoding performance across all conditions. However, offline posthoc analysis using subject-independent decoders revealed that AAD accuracy in the CLA condition was significantly lower than in the OL baseline. Subjectively, participants reported that CLA was significantly more distracting and required higher switching effort. Crucially, a causal analysis of the psCLA condition found no robust evidence that higher audio gains inherently improve decoding accuracy. Our results demonstrate that within a single-session paradigm with rapidly varying feedback cues, auditory neurofeedback may degrade AAD performance by increasing cognitive load and distraction. These findings suggest that suboptimal feedback can impede rather than facilitate learning. We conclude that more accurate and stable decoders and longitudinal, multi-session training protocols are likely essential prerequisites for achieving beneficial neurofeedback effects in closed-loop auditory attention systems.

neuroscience↗

bioRxiv2025

EEG-based Decoding of Auditory Attention to Conversations with Turn-taking Speakers

ObjectivesAuditory attention decoding (AAD) refers to the process of identifying which sound source a listener is attending to, based on neural recordings, such as electroencephalography (EEG). Most AAD studies use a competing speaker paradigm where two continuously active speech signals are simultaneously presented, in which the participant is instructed to attend to one speaker and ignore the other speaker. However, such a competing two-speaker scenario is uncommon in real life, as speakers typically take turns rather than speaking simultaneously. In this paper, we argue that decoding attention to conversations (rather than individual speakers) is a more relevant paradigm for testing AAD algorithms. In such a conversation-tracking paradigm, the AAD algorithm focusses on switching between entire conversations, resulting in less frequent attention shifts (ignoring turn-taking within conversations), thereby allowing for more relaxed constraints on the decision time. DesignTo test AAD performance in such a conversation-tracking paradigm, we simulated a challenging restaurant scenario with three simultaneous two-speaker conversations, which were podcasts presented in front of the listener and in the back left and back right of the room. We conducted an EEG experiment on 20 normal-hearing participants to compare the performance of AAD in the commonly used competing speaker paradigm with two speakers versus the conversation tracking paradigm with 2 or 3 conversations, each containing two turn-taking speakers. ResultsWe found that AAD, using stimulus decoding, worked well under all experimental conditions, and that the accuracy was not influenced by the direction of attention, the proximity to the target conversation, or the presence of within-trial attention switches (versus a condition with sustained attention). Given the challenging scenario, we probed for the participants listening experience and found a correlation between the neural decoding performance and the perceived listening effort and self-reported speech intelligibility. To gain insight into the speech intelligibility of the participants in our setup, they performed a speech-in-noise test (Flemish matrix sentence test), but we did not find a correlation between the speech intelligibility performance and the AAD performance.

neuroscience↗

bioRxiv2025

Neural Tracking of Audiovisual Effects in Noise Using Deep Neural Network-Generated Virtual Humans

This study investigates the effectiveness of Deep Neural Network (DNN)-generated virtual humans in enhancing audiovisual speech perception in noisy environments, with a focus on using neural measures to quantify these effects. Lip movements are essential for speech comprehension, especially when auditory cues are degraded by noise. Traditional recording methods produce high-quality audiovisual materials but are resource intensive. This research explores the use of DNN avatars as a promising alternative, utilizing a commercially available tool to create realistic virtual humans. The study included both simple sentences and a short story to improve ecological validity. Eleven young, normal-hearing participants proficient in Flemish-Dutch listened to semantically meaningful sentences and a short story with various speaker types: a female FACS avatar, male and female DNN avatars, and a video of a human male speaker. The study included behavioral measures which consisted of an adaptive recall procedure and an adaptive rate procedure and electrophysiological measures consisting of neural tracking. Findings in the adaptive recall procedure showed consistent audiovisual benefits, with the human speaker offering the greatest benefit (-4.75 dB SNR), followed by the DNN avatar (-4.00 dB SNR) and the FACS avatar (-1.55 dB SNR). Additionally in the adaptive rate procedure, the DNN avatar improved speech intelligibility, with average SRTs enhancing from -7.17 dB SNR (audio-only) to -9.02 dB SNR (audiovisual). The results from the neural tracking procedure indicated that most participants experienced audiovisual benefits, particularly in the -9 dB SNR range, revealing that audiovisual cues provided by DNN avatars can enhance speech perception, validating these avatars as effective tools for studying audiovisual effects when using both behavioral measures and electrophysiological measures.

neuroscience↗

bioRxiv2025

Effects of speaker and listener sex on auditory attention decoding performance

ObjectivesAuditory Attention Decoding (AAD) is a technique utilizing brain signals to decode on which sound the listener focuses the attention. In most current studies, the effect of type of speech materials used and sex of the listener is not considered. We investigated the effect on AAD performance of factors related to the speaker (such as the sex of the speaker, background noise level, and same versus mixed-sex conditions) and the listener (sex of the listener). DesignForty-two young adults with normal hearing participated in the study. They listened to 2 competing speakers and were instructed to attend to one speaker and ignore the other speaker, whilst electroencephalography (EEG) and electrooculography (EOG) were measured. Background noise was introduced in half of the conditions. AAD performance was compared across eight experimental conditions. ResultsA significant main effect of speaker sex was found: A male target and/or male masker speaker resulted in higher AAD performance compared to a female speaker with a higher fundamental frequency. These effects were found to be small and therefore likely clinically irrelevant. ConclusionWhile no substantial effects were found on the factors investigated in this study, including diverse and realistic training scenarios remains a valuable approach to prevent potential influences from other factors.

neuroscience↗

bioRxiv2025

Comparison of Neural Tracking and Spectral Entropy in Patients with Disorders of Consciousness

ObjectivesThis study aims to explore the brain responses of patients with disorders of consciousness (DoC) to natural speech. Specifically, it focuses on a key characteristic of natural speech: the speech envelope. To achieve this, we employed two distinct measures. The first evaluates how effectively the patients brain activity "tracks" the speech envelope, called neural tracking. The second assesses the complexity of the brains responses to the speech stimulus, measured through spectral entropy. These two measures are then compared in their association with the patients clinical diagnosis and their level of behavioral responsiveness, both of which were assessed using the Coma Recovery Scale-Revised (CRS-R). DesignFour patients with DoC participated in this study, during which their brain activity was recorded using electroen-cephalography (EEG). At the same time, they listened to a narrated story in both Dutch and Swedish. Additionally, EEG baseline recordings were collected. We employed a backward modeling approach to evaluate the speech envelopes neural tracking. This technique involves training a model to map the relationship between EEG signals and the corresponding speech envelope. Once the model is trained, it can use unseen EEG data to reconstruct the speech envelope, which is then compared to the original speech envelope to assess how effectively the patient processed the auditory stimulus. For the behavioral assessment, we recalculated the CRS-R score of each patient into the CSR-R index, a more meaningful score that utilizes all the information contained in the CRS-R instead of only the highest scores on each subscale. ResultsOur findings revealed positive correlations between spectral entropy and the CRS-R index, which were more pronounced during the listening conditions than the baseline. While neural tracking of the speech envelope did not correlate with the CRS-R index, it did exhibit a positive association with CRS-R diagnoses, indicating that patients with better clinical diagnoses demonstrated higher levels of neural tracking. Additionally, we identified an interaction effect between spectral entropy and neural tracking. Specifically, higher levels of neural tracking were associated with a stronger positive relationship between spectral entropy and the CRS-R index. In contrast, when neural tracking was lower, this relationship disappeared. ConclusionThis study demonstrated the potential of neural tracking and spectral entropy as complementary tools to investigate patients with DoC. Spectral entropy proved valuable for assessing behavioral responsiveness, while neural tracking shows promise in assessing the DoC diagnosis. Terms: disorders of consciousness (DoC), neural tracking, speech envelope, spectral entropy

neuroscience↗

bioRxiv2025

Large Language Models Reveal the Neural Tracking of Linguistic Context in Attended and Unattended Multi-Talker Speech

Large language models (LLMs) capture long-range contextual structure in natural language and have recently been shown to align with the human brains contextualized linguistic encoding. This makes them a promising computational probe for studying how context-dependent linguistic information is represented during natural speech perception. Speech perception often occurs in multi-talker environments, where attention must dynamically select among competing streams, yet how contextual information from attended and unattended speech is neurally encoded remains underexplored. Here, we investigate how auditory attention modulates neural tracking of context-dependent linguistic representations using electrocorticography (ECoG) and stereoelectroencephalography (sEEG) recordings from three epilepsy patients engaged in a two-conversation "cocktail party" paradigm. To model neural responses to attended and unattended speech streams, we used contextual word embeddings generated by large language models. We find that LLM-derived features reliably predict brain activity for the attended stream and that contextual information from the unattended stream also contributes to neural prediction. Importantly, these contributions extend beyond low-level acoustic features and shallow syntactic information, and depend on the surrounding linguistic context. Moreover, neural tracking of the unattended stream reflects shorter-range contextual integration than that of the attended stream. Together, these findings indicate that neural responses to speech reflect context-dependent linguistic representations from multiple concurrent speech streams, with attention modulating the depth and timescale of contextual integration. Our results highlight the utility of LLMs for probing higher-level linguistic representations in complex, naturalistic listening environments.

neuroscience↗

bioRxiv2025

Interpersonal synchronization as an objective measure of listening engagement

Listening engagement plays a crucial role in effective communication and knowledge transfer, reflecting a state of deep absorption in an auditory stimulus. The available behavioral assessments of listening engagement are limited in capturing its multidimensional nature. Measuring listening engagement objectively using brain and body signals is a compelling alternative. In particular, interpersonal synchronization--observed through synchronized neural and physiological responses among listeners--may offer a promising objective measure. In this study, we examined synchronization of brain activity, heart rate, and electrodermal activity between participants while they listened to engaging and non-engaging stories. Subjective engagement ratings were collected as a ground truth reference. Results showed higher interpersonal synchronization across all modalities when participants listened to engaging stories compared to non-engaging ones, with significant correlations between synchronization measures and ratings of subjective engagement. These results confirm that synchronization may be a reliable, objective marker of listening engagement. We anticipate that these results will provide a powerful framework for future research in relevant domains, including auditory neuroscience, education, and communication science, and enable impactful real-world applications, such as automatically tracking audience engagement in education, entertainment, and broader societal contexts.

neuroscience↗

bioRxiv2024

Classifying native versus foreign speech perception from EEG using linguistic speech features

When a person listens to natural speech, the relation between features of the speech signal and the corresponding evoked electroencephalogram (EEG) is indicative of neural processing of the speech signal. Using linguistic representations of speech, we investigate the differences in neural processing between speech in a native and foreign language that is not understood. We conducted experiments using three stimuli: a comprehensible language, an incomprehensible language, and randomly shuffled words from a comprehensible language, while recording the EEG signal of native Dutch-speaking participants. We modeled the neural tracking of linguistic features of the speech signals using a deep-learning model in a match-mismatch task that relates EEG signals to speech, while accounting for lexical segmentation features reflecting acoustic processing. The deep learning model effectively classifies languages. We also observed significant differences in tracking patterns between comprehensible and incomprehensible speech stimuli within the same language. It demonstrates the potential of deep learning frameworks in measuring speech understanding objectively.

neuroscience↗

bioRxiv2024

Neural substrates and behavioral relevance of speech envelope tracking: evidence from post-stroke aphasia

Neural tracking of the low-frequency temporal envelope of speech has emerged as a prominent tool to investigate the neural mechanisms of natural speech processing in the brain. However, there is ongoing debate regarding the functional role of neural envelope tracking. In this context, our study aims to offer a novel perspective by investigating the critical brain areas and behavioral skills required for neural envelope tracking in aphasia, a language disorder characterized by impaired neural envelope tracking. We analyzed an EEG dataset of 39 individuals with post-stroke aphasia suffering a left-hemispheric stroke who listened to natural speech. Our analysis involved lesion mapping, where left lesioned brain voxels served as binary features to predict neural envelope tracking measures. We also examined the behavioral correlates of receptive language, naming, and auditory processing (via rise time discrimination task) skills. The lesion mapping analysis revealed that lesions in language areas, such as the middle temporal gyrus, supramarginal gyrus and angular gyrus, were associated with poorer neural envelope tracking. Additionally, neural tracking was related to auditory processing skills and language (receptive and naming) skills. However, the effects on language skills were less robust, possibly due to ceiling effects in the language scores. Our [fi]ndings highlight the importance of central brain areas implicated in language understanding, extending beyond the primary auditory cortex, and emphasize the role of intact auditory processing and language abilities in effectively processing the temporal envelope of speech. Collectively, these [fi]ndings underscore the signi[fi]cance of neural envelope tracking beyond mere audibility and acoustic processes. Signi[fi]cance statementWhile some studies have proposed that neural envelope tracking primarily relates to audibility and acoustic speech processes, others have suggested its involvement in actual speech and language comprehension. By investigating the critical brain areas and behavioral skills essential in aphasia, we argue for a broader signi[fi]cance of neural envelope tracking in language processing. Furthermore, our [fi]ndings highlight a speci[fi]city among individuals with aphasia, indicating its correlation with lesions in temporal brain regions associated with receptive language functions. This addresses the signi[fi]cant heterogeneity in lesion characteristics present among individuals with aphasia and suggests the potential of neural tracking as an EEG-based tool for speci[fi]cally assessing receptive language abilities in this population.

neuroscience↗

bioRxiv2023

'Are you even listening?' - EEG-based detection of absolute auditory attention to natural speech with application to neuro-steered hearing devices

ObjectiveIn this study, we use electroencephalography (EEG) recordings to determine whether a subject is actively listening to a presented speech stimulus. More precisely, we aim to discriminate between an active listening condition, and a distractor condition where subjects focus on an unrelated distractor task while being exposed to a speech stimulus. We refer to this task as absolute auditory attention decoding. ApproachWe re-use an existing EEG dataset where the subjects watch a silent movie as a distractor condition, and introduce a new dataset with two distractor conditions (silently reading a text and performing arithmetic exercises). We focus on two EEG features, namely neural envelope tracking (NET) and spectral entropy (SE). Additionally, we investigate whether the detection of such an active listening condition can be combined with a selective auditory attention decoding task, where the goal is to decide to which of multiple competing speakers the subject is attending. The latter is a key task in so-called neuro-steered hearing devices that aim to suppress unattended audio, while preserving the attended speaker. Main resultsContrary to a previous hypothesis of higher SE being related with actively listening rather than passively listening (without any distractors), we find significantly lower SE in the active listening condition compared to the distractor conditions. Nevertheless, the NET is consistently significantly higher when actively listening. Similarly, we show that the accuracy of a selective auditory attention decoding task improves when evaluating the accuracy only on the highest NET segments. However, the reverse is observed when evaluating the accuracy only on the lowest SE segments. SignificanceWe conclude that the NET is more reliable for decoding absolute auditory attention as it is consistently higher when actively listening, whereas the relation of the SE between actively and passively listening seems to depend on the nature of the distractor.

neuroscience↗

bioRxiv2023

SparrKULee: A Speech-evoked Auditory ResponseRepository of the KU Leuven, containing EEG of 85participants

Researchers investigating the neural mechanisms underlying speech perception often employ electroencephalography (EEG) to record brain activity while participants listen to spoken language. The high temporal resolution of EEG enables the study of neural responses to fast and dynamic speech signals. Previous studies have successfully extracted speech characteristics from EEG data and, conversely, predicted EEG activity from speech features. Machine learning techniques are generally employed to construct encoding and decoding models, which necessitate a substantial amount of data. We present SparrKULee: A Speech-evoked Auditory Repository of EEG, measured at KU Leuven, comprising 64-channel EEG recordings from 85 young individuals with normal hearing, each of whom listened to 90-150 minutes of natural speech. This dataset is more extensive than any currently available dataset in terms of both the number of participants and the amount of data per participant. It is suitable for training larger machine learning models. We evaluate the dataset using linear and state-of-the-art non-linear models in a speech encoding/decoding and match/mismatch paradigm, providing benchmark scores for future research.

neuroscience↗

bioRxiv2023

EEG-based decoding of the spatial focus of auditory attention in a multi-talker audiovisual experiment using Common Spatial Patterns

ObjectiveSpatial auditory attention decoding (Sp-AAD) refers to the task of identifying the direction of the speaker to which a person is attending in a multi-talker setting, based on the listeners neural recordings, e.g., electroencephalography (EEG). The goal of this study is to thoroughly investigate potential biases when training such Sp-AAD decoders on EEG data, particularly eye-gaze biases and latent trial-dependent confounds, which may result in Sp-AAD models that decode eye-gaze or trial-specific fingerprints rather than spatial auditory attention. ApproachWe designed a two-speaker audiovisual Sp-AAD protocol in which the spatial auditory and visual attention were enforced to be either congruent or incongruent, and we recorded EEG data from sixteen participants undergoing several trials recorded at distinct timepoints. We trained a simple linear model for Sp-AAD based on common spatial patterns (CSP) filters in combination with either linear discriminant analysis (LDA) or k-means clustering, and evaluated them both across- and within-trial. Main resultsWe found that even a simple linear Sp-AAD model is susceptible to overfitting to confounding signal patterns such as eye-gaze and trial fingerprints (e.g., due to feature shifts across trials), resulting in artificially high decoding accuracies. Furthermore, we found that changes in the EEG signal statistics across trials deteriorate the trial generalization of the classifier, even when the latter is retrained on the test trial with an unsupervised algorithm. SignificanceCollectively, our findings confirm that there exist subtle biases and confounds that can strongly interfere with the decoding of spatial auditory attention from EEG. It is expected that more complicated non-linear models based on deep neural networks, which are often used for Sp-AAD, are even more vulnerable to such biases. Future work should perform experiments and model evaluations that avoid and/or control for such biases in Sp-AAD tasks.

neuroscience↗

bioRxiv2023

Neural envelope tracking predicts speech intelligibility and hearing aid benefit in children with hearing loss

Early assessment of hearing aid benefit is crucial, as the extent to which hearing aids provide audible speech information predicts speech and language outcomes. A growing body of research has proposed neural envelope tracking as an objective measure of speech, particularly for individuals unable to provide reliable behavioral feedback. However, its potential for evaluating speech intelligibility and hearing aid benefit in hearing-impaired children remains unexplored. This study examined neural envelope tracking in hearing-impaired children through two separate experiments. EEG data was recorded while children listened to age-appropriate stories (experiment 1) or an animated movie (experiment 2) under aided and unaided conditions (using personal hearing aids) at multiple stimulus intensities. Results in the delta band demonstrated that neural tracking increased with increasing stimulus intensity, but only in the unaided condition. In the aided condition, neural tracking remained stable across a wide range of intensities, as long as speech intelligibility was maintained. This suggests that acoustic degradation of the speech signal does not necessarily impact neural tracking. Additionally, the use of personal hearing aids significantly enhanced neural envelope tracking, particularly in challenging speech conditions (which would be inaudible when unaided). Furthermore, neural envelope tracking strongly correlated with behaviorally measured speech intelligibility. Altogether, these findings indicate that neural envelope tracking could be a valuable tool for predicting speech intelligibility benefits derived from personal hearing aids in hearing-impaired children. Incorporating narrated stories or engaging movies expands the accessibility of these methods even in clinical settings, offering new avenues for using objective speech measures to guide pediatric audiology decision-making.

neuroscience↗

bioRxiv2023

Auditory steady-state responses: multiplexed amplitude modulation frequencies to reduce recording time

ObjectivesThis study investigated the efficiency of a multiplexed amplitude-modulated (AM) stimulus in eliciting auditory steady-state responses (ASSR). The multiplexed AM stimulus was created by simultaneously modulating speech-shaped noise with three frequencies chosen to elicit different neural generators: 3.1, 40.1, and 102.1 Hz. For comparison, a single AM stimulus was created for each of these frequencies, resulting in three single AM conditions and one multiplex AM condition. DesignTwenty-two bilaterally normal-hearing participants (18 female) listened for 8 minutes to each type of stimuli. The analysis compared the signal-to-noise ratios (SNRs) and amplitudes of the evoked responses to the single and multiplexed conditions. ResultsThe results revealed that the SNRs elicited by single AM conditions were, on average, 1.61 dB higher than those evoked by the multiplexed AM condition (p < 0.05). The single conditions consistently produced a significantly higher SNR when examining various stimulus durations ranging from 1 to 8 minutes. Despite these SNR differences, the frequency spectrum was very similar across and within subjects. Additionally, the sensor space patterns across the scalp demonstrated similar trends between the single and multiplexed stimuli for both SNR and amplitudes. Both the single and multiplexed conditions evoked significant ASSRs within subjects. On average, the multiplexed AM stimulus took 31 minutes for the lower bound of the 95% prediction interval to cross the significance threshold across all three frequencies. In contrast, the single AM stimuli took 45 minutes and 42 seconds. ConclusionThese findings show that the multiplexed AM stimulus is a promising method to reduce the recording time when simultaneously obtaining information from various neural generators.

neuroscience↗

bioRxiv2023

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker's mouth is occluded

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

neuroscience↗

bioRxiv2023

Robust neural tracking of linguistic speech representations using a convolutional neural network.

ObjectiveWhen listening to continuous speech, populations of neurons in the brain track different features of the signal. Neural tracking can be measured by relating the electroencephalography (EEG) and the speech signal. Recent studies have shown a significant contribution of linguistic features over acoustic neural tracking using linear models. However, linear models cannot model the nonlinear dynamics of the brain. To overcome this, we use a convolutional neural network (CNN) that relates EEG to linguistic features using phoneme or word onsets as a control and has the capacity to model non-linear relations. ApproachWe integrate phoneme- and word-based linguistic features (phoneme surprisal, cohort entropy, word surprisal and word frequency) in our nonlinear CNN model and investigate if they carry additional information on top of lexical features (phoneme and word onsets). We then compare the performance of our nonlinear CNN with that of a linear encoder and a linearized CNN. Main resultsFor the non-linear CNN, we found a significant contribution of cohort entropy over phoneme onsets and of word surprisal and word frequency over word onsets. Moreover, the non-linear CNN outperformed the linear baselines. SignificanceMeasuring coding of linguistic features in the brain is important for auditory neuroscience research and applications that involve objectively measuring speech understanding. With linear models, this is measurable, but the effects are very small. The proposed non-linear CNN model yields larger differences between linguistic and lexical models and, therefore, could show effects that would otherwise be unmeasurable and may, in the future, lead to improved within-subject measures and shorter recordings. Index TermsEEG decoding, speech processing, CNN, linguistics.

neuroscience↗