Biology subjects

Cao, C. Z.

Publications and source records attributed to Cao, C. Z..

4 recordsLinked to original sources

bioRxiv2024

Visual speech enhances auditory onset timing and envelope tracking through distinct mechanisms

Seeing a speakers face facilitates speech recognition in challenging listening environments. Prior work has shown that visual speech contains timing information to aid auditory speech processing, yet how these signals are integrated within the auditory system during audiovisual speech perception remains poorly understood. Observation of preparatory mouth movements may initiate phase reset of intrinsic oscillations, potentially sensitizing the auditory system for receptive speech processing, while observation of mouth movements post speech onset may facilitate entrainment to the speech envelope. Yet, little work has been done to test whether visual speech enhances encoding of auditory speech onset, speech envelope tracking, or both, and through independent or overlapping mechanisms. To investigate this, we examined the ways in which visual speech timing information alters theta band power and phase using human intracranial electroencephalography (iEEG) recordings in a large group of patients with epilepsy (n = 21). Prior to speech onset, preparatory mouth movements elicited theta phase reset (increased inter-trial phase coherence; ITPC) throughout the superior temporal gyrus (STG), which is thought to enhance speech onset encoding. Following speech onset, visual speech modulated theta ITPC only at anterior STG electrodes while theta power was modulated at posterior STG electrodes. Pre- and post-speech onset were spatially and temporally dissociated, consistent with the hypothesis that audiovisual speech onset encoding and envelope tracking mechanisms are partially distinct. Crucially, congruent and incongruent visual speech, designed here to have identical visual timing information about speech onset time, but different visual mouth evolution, produced only a small difference in the phase of theta band oscillations in the anterior STG, highlighting a more restricted role of visual speech in ongoing auditory entrainment. These results support the hypothesis that visual speech improves the precision of auditory speech encoding through two separate mechanisms, with auditory speech onset encoded throughout the entire STG and ongoing speech envelope tracking within anterior STG.

neuroscience↗

bioRxiv2022

Auditory cortex encodes lipreading information through spatially distributed activity

Watching a speakers face improves speech perception accuracy. These benefits are owed, in part, to implicit lipreading abilities present in the general population. While it is established that lipreading can alter the perception of a heard word, it is unknown how information that is extracted from lipread words is transformed into a neural code that the auditory system can use. One influential, but untested, hypothesis is that visual speech modulates the population coded representations of phonetic and phonemic features in the auditory system. This model is largely supported by data showing that silent lipreading evokes activity in auditory cortex, but these activations could alternatively reflect general effects of arousal or attention, or the encoding of non-linguistic features such as visual timing information. This gap limits our understanding of how vision supports speech perception processes. To test the hypothesis that the auditory system encodes visual speech information, we acquired fMRI data from healthy adults and intracranial recordings from electrodes implanted in patients with epilepsy during auditory and visual speech perception tasks. Across both methods, linear classifiers successfully decoded the identity of silently lipread words using the spatial pattern of auditory cortex responses. Examining the time-course of classification using intracranial recordings, lipread words were classified at significantly earlier time-points relative to heard words, suggesting a predictive mechanism for facilitating speech. These results support a model in which the auditory system combines the joint neural distributions evoked by heard and lipread words to generate a more precise estimate of what was said. Significance StatementWhen we listen to someone speak in a noisy environment, watching their face can help us understand them better, largely due to automatic lipreading abilities. However, it unknown how lipreading information is transformed into a neural code that the auditory system can use. We used fMRI and intracranial recordings in patients to study how the brain processes silently lipread words and found that the auditory system encodes the identity of lipread words through spatially distributed activity. These results suggest that the auditory system combines information from both lipreading and hearing to generate more precise estimates of what is said, potentially by both activating the corresponding representation of the heard word and suppressing incorrect phonemic representations.

neuroscience↗

bioRxiv2022

The strength of feedback processing is associated with resistance to visual backward masking during illusory contour processing in adult humans.

Re-entrant feedback processing is a key mechanism of visual object-recognition, especially under compromised viewing conditions where only sparse information is available and object features must be interpolated. Illusory contour stimuli are commonly used in conjunction with visual evoked potentials (VEP) to study these filling-in processes, with characteristic modulation of the VEP in the ~100-150ms timeframe associated with this re-entrant processing. Substantial inter-individual variability in timing and amplitude of feedback-related VEP modulation is observed, raising the question whether this variability might underlie inter-individual differences in the ability to form strong perceptual gestalts. Backward masking paradigms have been used to study inter-individual variance in the ability to form robust object perceptions before processing of the mask interferes with object-recognition. Some individuals recognize objects when the time between target object and mask is extremely short, whereas others struggle to do so even at longer target-to-mask intervals. We asked whether timing and amplitude of feedback-related VEP modulations were associated with individual differences in resistance to backward masking. Participants (N=40) showed substantial performance variability in detecting illusory contours at intermediate target-to-mask intervals (67ms and 117ms), allowing us to use kmeans clustering to divide the population into four performance groups (poor, low-average, high-average, superior). There was a clear relationship between the amplitude (but not the timing) of feedback-related VEP modulation and illusory contour detection during backward masking. We conclude that individual differences in the strength of feedback processing in neurotypical humans lead to differences in the ability to quickly establish perceptual awareness of incomplete visual objects.

neuroscience↗

bioRxiv2021

A linguistic representation in the visual system underlies successful lipreading

There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Surprisingly, much of this ambiguity stems from a lack of behaviorally grounded neurophysiological findings. To address this, we conducted an experiment in which human observers rehearsed audiovisual speech for the purpose of lipreading silent versions during testing. Using a combination of computational modeling, electroencephalography, and simultaneously recorded behavior, we show that the visual system produces its own specialized representation of speech that is 1) well-described by categorical linguistic units ("visemes") 2) dissociable from lip movements, and 3) predictive of lipreading ability. These findings contradict a long-held view that visual speech processing co-opts auditory cortex after early visual processing stages. Consistent with hierarchical accounts of visual and audiovisual speech perception, our findings show that visual cortex performs at least a basic level of linguistic processing.

neuroscience↗