Biology subjects

Magnotti, J. F.

Publications and source records attributed to Magnotti, J. F..

6 recordsLinked to original sources

bioRxiv2025

A Deep Neural Network Trained on Congruent Audiovisual Speech Reports the McGurk Effect

In the McGurk effect, perception of an auditory syllable changes dramatically when it is paired with an incongruent visual syllable, countering our intuition that speech perception is solely an auditory process. The dominant modeling framework for the study of audiovisual speech perception is that of Bayesian causal inference, but current Bayesian models are unable to predict the wide range of percepts evoked by McGurk syllables. We explored whether a deep neural network (DNN) known as AVHuBERT could provide an alternative modeling framework. AVHuBERT model variants were presented with McGurk syllables consisting of auditory "ba" paired with visual "ga" recorded from 8 different talkers. AVHuBERT identified McGurk syllables as something other than "ba" at a rate of 59%, demonstrating a robust McGurk effect. The rate of the McGurk effect was similar to that observed in humans: 100 participants presented with the same McGurk syllables reported non-"ba" percepts on 56% of trials. AVHubert variants and humans produced a wide variety of responses to McGurk syllables, including the canonical McGurk fusion percept of "da", responses without any initial consonant such as "ah" and responses with other initial consonants such as "fa". The ability to predict percepts experienced by humans but not predicted by current Bayesian models suggest that DNNs and Bayesian models may provide complementary windows into the perceptual mechanisms underlying human audiovisual speech perception.

neuroscience↗

bioRxiv2024

Integrating Data Across Oscillatory Power Bands Predicts the Epileptogenic Zone: the Frequency Range Explorer Epileptogenic Zone (FREEZ) Identification Algorithm

Accurate identification of the seizure onset zone (SOZ) using intracranial electroencephalography (iEEG) remains challenging. Although diverse methods have leveraged spectral features to classify patient outcomes, few approaches focus on identifying individual electrodes within the SOZ or integrate a broad spectrum of frequency ranges within a single model. We developed an interpretable machine learning model that integrates power across delta, theta, alpha, beta, gamma, and high-gamma frequencies over time to identify the SOZ. For 1,511 electrodes implanted across 21 patients, we computed the mean spectral power in each frequency band for the first 20 seconds after seizure onset and analyzed the differences in power between SOZ and non-SOZ electrodes. In patients who were seizure-free after surgery (n = 14), electrodes within the SOZ showed significantly higher area under the curve (AUC) for mean power over time in the first 20 seconds after seizure onset compared to electrodes outside the SOZ in the alpha (p = 0.0272), beta (p = 0.0263), gamma (p = 0.0013), and high gamma (p = 0.0086) ranges. Additionally, electrodes within the SOZ in patients that became seizure-free after surgery had significantly higher AUC compared to equivalent electrodes in patients who did not become seizure-free after surgery (n = 7) in the gamma (p = 0.0145) and high gamma (p = 0.0024) power ranges. We trained a stacked random forest ensemble model using these features over time to label electrodes within the SOZ. Leave-one-out patient cross validation of the machine learning model yielded a 96.6% positive predictive value and 99.9% specificity for identifying electrodes within the SOZ. Our dataset included a diverse array of seizure onset patterns, which were all classified accurately by the model. A second model was trained to predict post-operative seizure freedom, yielding 95.2% accuracy for predicting seizure outcome based on a planned resection. This two-model design mirrors clinical workflow, first localizing SOZ electrodes to support surgical planning, then predicting outcome based on a surgical plan. An advantage of our interpretable machine learning approach is the ability to interrogate our models to understand how predictions are made. For electrode classification, the model weighed beta (0.66 {+/-} 0.07), high gamma (0.54 {+/-} 0.06), and delta (0.51 {+/-} 0.06) power bands most heavily. Viewing the models frequency band weights over time reveals that the model identified a pattern resembling the "fingerprint of the epileptogenic zone", reinforcing the importance of this dominant fundamental neurophysiologic signature of seizure onset.

neuroscience↗

bioRxiv2024

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking face generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the real vs. synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces, providing a roadmap for improving communication in this rapidly developing domain.

neuroscience↗

bioRxiv2020

Causal inference explains the stimulus-level relationship between the McGurk Effect and auditory speech perception

The McGurk effect is widely used as a measure of multisensory integration during speech perception. Two observations have raised questions about the relationship between the effect and everyday speech perception. First, there is high variability in the strength of the McGurk effect across different stimuli and observers. Second, there is low correlation across observers between perception of the McGurk effect and measures of everyday speech perception, such as the ability to understand noisy audiovisual speech. Using the framework of the causal inference of multisensory speech (CIMS) model, we explored the relationship between the McGurk effect, syllable perception, and sentence perception in seven experiments with a total of 296 different participants. Perceptual reports revealed a relationship between the efficacy of different McGurk stimuli created from the same talker and perception of the auditory component of the McGurk stimuli presented in isolation, either with or without added noise. The CIMS model explained this high stimulus-level correlation using the principles of noisy sensory encoding followed by optimal cue combination within a representational space that was identical for McGurk and everyday speech. In other experiments, CIMS successfully modeled low observer-level correlation between McGurk and everyday speech. Variability in noisy speech perception was modeled using individual differences in noisy sensory encoding, while variability in McGurk perception involved additional differences in causal inference. Participants with all combinations of high and low sensory encoding noise and high and low causal inference disparity thresholds were identified. Perception of the McGurk effect and everyday speech can be explained by a common theoretical framework that includes causal inference.

neuroscience↗

bioRxiv2019

Principles of tactile search over the body

Although we experience complex patterns over our entire body, how we selectively perceive multi-site touch over our bodies remains poorly understood. Here, we characterized tactile search behavior over the body using a tactile analog of the classic visual search task. Participants judged whether a target stimulus (e.g., 10-Hz vibration) was present or absent on the upper or lower limbs. When present, the target stimulus could occur alone or with distractor stimuli (e.g., 30-Hz vibrations) on other body locations. We varied the number and spatial configurations of the distractors as well as the target and distractor frequencies and measured the impact of these factors on search response times. First, we found that response times were faster on target-present trials compared to target-absent trials. Second, response times increased with the number of stimulated sites, suggesting a serial search process. Third, search performance differed depending on stimulus frequencies. This frequency-dependent behavior may be related to perceptual grouping effects based on timing cues. We constructed models to explore how the locations of the tactile cues influenced search behavior. Our modeling results reveal that, in isolation, cues on the index fingers make relatively greater contributions to search performance compared to stimulation experienced on other body sites. Additionally, co-stimulation of sites within the same limb or simply on the same body side preferentially influence search behavior. Our collective findings identify some principles of attentional search that are common to vision and touch, but others that highlight key differences that may be unique to body-based spatial perception. New & NoteworthyLittle is known about how we selectively experience multi-site touch over the body. Using a tactile analog of the classic visual search paradigm, we show that tactile search behavior for flutter cues is generally consistent with a serial search process. Modeling results reveal the preferential contributions of index finger stimulation and two-site interactions involving ipsilateral and within-limb patterns. Our results offer initial evidence for spatial and temporal principles underlying tactile search behavior over the body.

neuroscience↗

bioRxiv2019

Cross-modal suppression of auditory association cortex by visual speech as a mechanism for audiovisual speech perception

Vision provides a perceptual head start for speech perception because most speech is \"mouth-leading\": visual information from the talkers mouth is available before auditory information from the voice. However, some speech is \"voice-leading\" (auditory before visual). Consistent with a model in which vision modulates subsequent auditory processing, there was a larger perceptual benefit of visual speech for mouth-leading vs. voice-leading words (28% vs. 4%). The neural substrates of this difference were examined by recording broadband high-frequency activity from electrodes implanted over auditory association cortex in the posterior superior temporal gyrus (pSTG) of epileptic patients. Responses were smaller for audiovisual vs. auditory-only mouth-leading words (34% difference) while there was little difference (5%) for voice-leading words. Evidence for cross-modal suppression of auditory cortex complements our previous work showing enhancement of visual cortex (Ozker et al., 2018b) and confirms that multisensory interactions are a powerful modulator of activity throughout the speech perception network.\n\nImpact StatementHuman perception and brain responses differ between words in which mouth movements are visible before the voice is heard and words for which the reverse is true.

physiology↗