Search bioRxiv⌕ Search

Biology subjects

Friedman, D.

Publications and source records attributed to Friedman, D..

At least 37 records · Page 2Linked to original sources

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals

ObjectiveThis study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. ApproachWe propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train subject-specific models using data from a single participant and multi-patient models exploiting data from multiple participants. Main ResultsThe subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. SignificanceThe proposed SwinTW decoder enables future speech neuropros-theses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests that such a model can be applied to new patients that do not have paired acoustic and neural data, providing an advance in neuroprostheses for people with speech disability, where acoustic-neural training data is not feasible.

neuroscience↗

Proliferating CLL cells express high levels of CXCR4 and CD5

Chronic Lymphocytic Leukaemia (CLL) is an incurable progressive malignancy of CD5+ B cells with a birth rate between 0.1-1% of the entire clone per day. However, the phenotype and functional characteristics of proliferating CLL cells remain incompletely understood. Here, we stained peripheral blood CLL cells for ki67 and DNA content and found that CLL cells in G1-phase have a CXCR4loCD5hi phenotype, whilst CLL cells in S/G2/M-phase express high levels of both CXCR4 and CD5. Induction of proliferation in vitro using CD40L stimulation results in high ki67 levels in CXCR4loCD5hi cells with CXCR4 expression increasing as CLL cells progress through S and G2/M-phases, whilst CXCR4hiCD5lo CLL cells remained quiescent. Dye dilution experiments revealed an accumulation of Ki67hi divided cells in the CXCR4hiCD5hi fraction. In E-TCL1 transgenic mice, the CXCR4hiCD5hi fraction expressed high levels of ki67 and was expanded in enlarged spleens of diseased animals. Human peripheral blood CXCR4hiCD5hi CLL cells express high levels of IgM and the chemokine receptors CCR7 and CXCR5 and migrated most efficiently towards CCL21. We found higher levels of CXCR4 in patients with progressive disease and the CXCR4hiCD5hi fraction was expanded upon clinical relapse. Thus, this study defines the phenotype and functional characteristics of proliferating CLL cells identifying a novel subclonal population that underlies CLL pathogenesis and may drive clinical outcome.

cancer biology↗

Speech-induced suppression and vocal feedback sensitivity in human cortex

Across the animal kingdom, neural responses in the auditory cortex are suppressed during vocalization, and humans are no exception. A common hypothesis is that suppression increases sensitivity to auditory feedback, enabling the detection of vocalization errors. This hypothesis has been previously confirmed in non-human primates, however a direct link between auditory suppression and sensitivity in human speech monitoring remains elusive. To address this issue, we obtained intracranial electroencephalography (iEEG) recordings from 35 neurosurgical participants during speech production. We first characterized the detailed topography of auditory suppression, which varied across superior temporal gyrus (STG). Next, we performed a delayed auditory feedback (DAF) task to determine whether the suppressed sites were also sensitive to auditory feedback alterations. Indeed, overlapping sites showed enhanced responses to feedback, indicating sensitivity. Importantly, there was a strong correlation between the degree of auditory suppression and feedback sensitivity, suggesting suppression might be a key mechanism that underlies speech monitoring. Further, we found that when participants produced speech with simultaneous auditory feedback, posterior STG was selectively activated if participants were engaged in a DAF paradigm, suggesting that increased attentional load can modulate auditory feedback sensitivity.

neuroscience↗

A Neural Speech Decoding Framework Leveraging Deep Learning and Speech Synthesis

Decoding human speech from neural signals is essential for brain-computer interface (BCI) technologies restoring speech function in populations with neurological deficits. However, it remains a highly challenging task, compounded by the scarce availability of neural signals with corresponding speech, data complexity, and high dimensionality, and the limited publicly available source code. Here, we present a novel deep learning-based neural speech decoding framework that includes an ECoG Decoder that translates electrocorticographic (ECoG) signals from the cortex into interpretable speech parameters and a novel differentiable Speech Synthesizer that maps speech parameters to spectrograms. We develop a companion audio-to-audio auto-encoder consisting of a Speech Encoder and the same Speech Synthesizer to generate reference speech parameters to facilitate the ECoG Decoder training. This framework generates natural-sounding speech and is highly reproducible across a cohort of 48 participants. Among three neural network architectures for the ECoG Decoder, the 3D ResNet model has the best decoding performance (PCC=0.804) in predicting the original speech spectrogram, closely followed by the SWIN model (PCC=0.796). Our experimental results show that our models can decode speech with high correlation even when limited to only causal operations, which is necessary for adoption by real-time neural prostheses. We successfully decode speech in participants with either left or right hemisphere coverage, which could lead to speech prostheses in patients with speech deficits resulting from left hemisphere damage. Further, we use an occlusion analysis to identify cortical regions contributing to speech decoding across our models. Finally, we provide open-source code for our two-stage training pipeline along with associated preprocessing and visualization tools to enable reproducible research and drive research across the speech science and prostheses communities.

neuroscience↗

Timing and location of speech errors induced by direct cortical stimulation

Cortical regions supporting speech production are commonly established using neuroimaging techniques in both research and clinical settings. However, for neurosurgical purposes, structural function is routinely mapped peri-operatively using direct electrocortical stimulation. While this method is the gold standard for identification of eloquent cortical regions to preserve in neurosurgical patients, there is lack of specificity of the actual underlying cognitive processes being interrupted. To address this, we propose mapping the temporal dynamics of speech arrest across peri-sylvian cortices by quantifying the latency between stimulation and speech deficits. In doing so, we are able to substantiate hypotheses about distinct region-specific functional roles (e.g., planning versus motor execution). In this retrospective observational study, we analyzed 20 patients (12 female; age range 14-43) with refractory epilepsy who underwent continuous extra-operative intracranial EEG monitoring of an automatic speech task during clinical bedside language mapping. Latency to speech arrest was calculated as time from stimulation onset to speech arrest onset, controlling for individual speech rate. Most instances of motor-based arrest (87.5% of 96 instances) were in sensorimotor cortex with mid-range latencies to speech arrest with a distributional peak at 0.47 seconds. Speech arrest occurred in numerous regions, with relatively short latencies in supramarginal gyrus (0.46 seconds), superior temporal gyrus (0.51 seconds), and middle temporal gyrus (0.54 seconds), followed by relatively long latencies in sensorimotor cortex (0.72 seconds) and especially long latencies in inferior frontal gyrus (0.95 seconds). Nonparametric testing for speech arrest revealed that region predicted latency; latencies in supramarginal gyrus and in superior temporal gyrus were shorter than in sensorimotor cortex and in inferior frontal gyrus. Sensorimotor cortex is primarily responsible for motor-based arrest. Latencies to speech arrest in supramarginal gyrus and superior temporal gyrus (and to a lesser extent middle temporal gyrus) align with latencies to motor-based arrest in sensorimotor cortex. This pattern of relatively quick cessation of speech suggests that stimulating these regions interferes with the outgoing motor execution. In contrast, the latencies to speech arrest in inferior frontal gyrus and in ventral regions of sensorimotor cortex were significantly longer than those in temporoparietal regions. Longer latencies in the more frontal areas (including inferior frontal gyrus and ventral areas of precentral gyrus and postcentral gyrus) suggest that stimulating these areas interrupts a higher-level speech production process involved in planning. These results implicate the ventral specialization of sensorimotor cortex (including both precentral and postcentral gyri) for speech planning above and beyond motor execution.

neuroscience↗

Temporal dynamics of neural adaptation across human visual cortex

Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventraland lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses across the human visual hierarchy and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences. Author summaryNeural responses in visual cortex adapt over time, with reduced responses to prolonged and repeated stimuli. Here, we examine how adaptation patterns differ across the visual hierarchy in neural responses recorded from human visual cortex with high temporal and spatial precision. To identify possible neural computations underlying short-term adaptation, we fit the response time courses using a temporal divisive normalization model. The model accurately predicts prolonged and repeated responses in lower and higher visual areas, and reveals differences in temporal adaptation across the visual hierarchy and stimulus categories. Our model suggests that differences in adaptation patterns result from differences in divisive normalization dynamics. Our findings shed light on how information is integrated in the brain on a millisecond-time scale and offer an intuitive framework to study the emergence of neural dynamics across brain areas and stimuli.

neuroscience↗

A shared linguistic space for transmitting our thoughts from brain to brain in natural conversations

Effective communication hinges on a mutual understanding of word meaning in different contexts. The embedding space learned by large language models can serve as an explicit model of the shared, context-rich meaning space humans use to communicate their thoughts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We demonstrate that the linguistic embedding space can capture the linguistic content of word-by-word neural alignment between speaker and listener. Linguistic content emerged in the speakers brain before word articulation, and the same linguistic content rapidly reemerged in the listeners brain after word articulation. These findings establish a computational framework to study how human brains transmit their thoughts to one another in real-world contexts.

neuroscience↗

Deep speech-to-text models capture the neural basis of spontaneous speech in everyday conversations

Humans effortlessly use the continuous acoustics of speech to communicate rich linguistic meaning during everyday conversations. In this study, we leverage 100 hours (half a million words) of spontaneous open-ended conversations and concurrent high-quality neural activity recorded using electrocorticography (ECoG) to decipher the neural basis of real-world speech production and comprehension. Employing a deep multimodal speech-to-text model named Whisper, we develop encoding models capable of accurately predicting neural responses to both acoustic and semantic aspects of speech. Our encoding models achieved high accuracy in predicting neural responses in hundreds of thousands of words across many hours of left-out recordings. We uncover a distributed cortical hierarchy for speech and language processing, with sensory and motor regions encoding acoustic features of speech and higher-level language areas encoding syntactic and semantic information. Many electrodes--including those in both perceptual and motor areas--display mixed selectivity for both speech and linguistic features. Notably, our encoding model reveals a temporal progression from language-to-speech encoding before word onset during speech production and from speech-to-language encoding following word articulation during speech comprehension. This study offers a comprehensive account of the unfolding neural responses during fully natural, unbounded daily conversations. By leveraging a multimodal deep speech recognition model, we highlight the power of deep learning for unraveling the neural mechanisms of language processing in real-world contexts.

neuroscience↗

Spatial Tuning of Alpha Oscillations in Human Visual Cortex

AO_SCPLOWBSTRACTC_SCPLOWNeuronal oscillations at about 10 Hz, called alpha oscillations, are often thought to arise from synchronous activity across occipital cortex, reflecting general cognitive states such as arousal and alertness. However, there is also evidence that modulation of alpha oscillations in visual cortex can be spatially specific. Here, we used intracranial electrodes in human patients to measure alpha oscillations in response to visual stimuli whose location varied systematically across the visual field. We separated the alpha oscillatory power from broadband power changes. The variation in alpha oscillatory power with stimulus position was then fit by a population receptive field (pRF) model. We find that the alpha pRFs have similar center locations to pRFs estimated from broadband power (70-180 Hz) but are several times larger. The results demonstrate that alpha suppression in human visual cortex can be precisely tuned. Finally, we show how the pattern of alpha responses can explain several features of exogenous visual attention. Significance StatementThe alpha oscillation is the largest electrical signal generated by the human brain. An important question in systems neuroscience is the degree to which this oscillation reflects system-wide states and behaviors such as arousal, alertness, and attention, versus much more specific functions in the routing and processing of information. We examined alpha oscillations at high spatial precision in human patients with intracranial electrodes implanted over visual cortex. We discovered a surprisingly high spatial specificity of visually driven alpha oscillations, which we quantified with receptive field models. We further use our discoveries about properties of the alpha response to show a link between these oscillations and the spread of visual attention. Grant support: NIH R01 MH111417 (Petridou, Winawer, Ramsey, Devinsky); JSPS Overseas Research Fellowship (Yuasa) The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health

neuroscience↗

A Corollary Discharge Circuit in Human Speech

When we vocalize, our brain distinguishes self-generated sounds from external ones. A corollary discharge signal supports this function in animals, however, in humans its exact origin and temporal dynamics remain unknown. We report Electrocorticographic (ECoG) recordings in neurosurgical patients and a novel connectivity approach based on Granger-causality that reveals major neural communications. We find a reproducible source for corollary discharge across multiple speech production paradigms localized to ventral speech motor cortex before speech articulation. The uncovered discharge predicts the degree of auditory cortex suppression during speech, its well-documented consequence. These results reveal the human corollary discharge source and timing with far-reaching implication for speech motor-control as well as auditory hallucinations in human psychosis. Significance statementHow do organisms dissociate self-generated sounds from external ones? A fundamental brain circuit across animals addresses this question by transmitting a blueprint of the motor signal to sensory cortices, referred to as a corollary discharge. However, in humans and non-human primates auditory system, the evidence supporting this circuit has been limited to its direct consequence, auditory suppression. Furthermore, an impaired corollary discharge circuit in humans can lead to auditory hallucinations. While hypothesized to originate in the frontal cortex, direct evidence localizing the source and timing of an auditory corollary discharge is lacking in humans. Leveraging rare human neurosurgical recordings combined with connectivity techniques, we elucidate the exact source and dynamics of the corollary discharge signal in human speech. One-sentence summaryWe reveal the source and timing of a corollary discharge from speech motor cortex onto auditory cortex in human speech.

neuroscience↗

Correspondence between the layered structure of deep language models and temporal structure of natural language processing in the human brain

Deep language models (DLMs) provide a novel computational paradigm for how the brain processes natural language. Unlike symbolic, rule-based models described in psycholinguistics, DLMs encode words and their context as continuous numerical vectors. These "embeddings" are constructed by a sequence of computations organized in "layers" to ultimately capture surprisingly sophisticated representations of linguistic structures. How does this layered hierarchy map onto the human brain during natural language comprehension? In this study, we used electrocorticography (ECoG) to record neural activity in language areas along the superior temporal gyrus and inferior frontal gyrus while human participants listened to a 30-minute spoken narrative. We supplied this same narrative to a high-performing DLM (GPT2-XL) and extracted the contextual embeddings for each word in the story across all 48 layers of the model. We next trained a set of linear encoding models to predict the temporally-evolving neural activity from the embeddings at each layer. We found a striking correspondence between the layer-by-layer sequence of embeddings from GPT2-XL and the temporal sequence of neural activity in language areas. In addition, we found evidence for the gradual accumulation of recurrent information along the linguistic processing hierarchy. However, we also noticed additional neural processes in the brain, but not in DLMs, during the processing of surprising (unpredictable) words. These findings point to a connection between human language processing and DLMs where the layer-by-layer accumulation of contextual information in DLM embeddings matches the temporal dynamics of neural activity in high-order language areas.

neuroscience↗

Hypothalamic-pituitary-adrenal axis dysfunction worsens epilepsy outcomes and increases SUDEP risk

Epilepsy is often comorbid with psychiatric illnesses, including anxiety and depression. Despite the high incidence of psychiatric comorbidities in people with epilepsy, few studies address the underlying mechanisms. Stress can trigger both epilepsy and depression and significant accumulating evidence, from both human and animal studies, support that hypothalamic-pituitary-adrenal (HPA) axis dysfunction may contribute to the comorbidity of these two disorders1. Here we directly investigate the contribution of HPA axis dysfunction to epilepsy outcomes and psychiatric comorbidities. We generated a novel mouse model (KCC2/Crh mice) that lacks the K+/Cl- co-transporter, KCC2, in corticotropin-releasing hormone (CRH) neurons, which exhibit stress- and seizure-induced hyperactivation of the HPA axis.2,3 Here we demonstrate that HPA axis dysfunction in KCC2/Crh mice is associated with increased vulnerability to behavioral deficits in chronically epileptic mice and more severe neuropathological features associated with chronic epilepsy (e.g. increased mossy fiber sprouting). Despite equivalent seizure burden, HPA axis dysfunction in these chronically epileptic mice increased the incidence of sudden unexpected death in epilepsy (SUDEP). Suppressing HPA axis hyperexcitability in this model using a pharmacological or chemogenetic approach decreased SUDEP incidence, supporting HPA axis dysfunction in SUDEP vulnerability. We also observed changes in neuroendocrine markers in individuals that died of SUDEP compared to people with epilepsy or individuals without a history of epilepsy. Together, these findings describe a novel, nongenetic mouse model of SUDEP which will benefit preclinical SUDEP research to gain a better understanding of the underlying pathological mechanisms contributing to SUDEP. HPA axis dysfunction increases vulnerability to psychiatric comorbidities in epilepsy and may be a risk factor for SUDEP.

neuroscience↗

Brain embeddings with shared geometry to artificial contextual embeddings, as a code for representing language in the human brain

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. Do language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language? To test this hypothesis, we densely recorded the neural activity in the Inferior Frontal Gyrus (IFG, also known as Brocas area) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derived for each patient a continuous vectorial representation for each word (i.e., a brain embedding). Using stringent, zero-shot mapping, we demonstrated that brain embeddings in the IFG and the DLM contextual embedding space have strikingly similar geometry. This shared geometry allows us to precisely triangulate the position of unseen words in both the brain embedding space (zero-shot encoding) and the DLM contextual embedding space (zero-shot decoding). The continuous brain embedding space provides an alternative computational framework for how natural language is represented in cortical language areas.

neuroscience↗

Distributed Feedforward and Feedback Processing across Perisylvian Cortex Supports Human Speech

Speech production is a complex human function requiring continuous feedforward commands together with reafferent feedback processing. These processes are carried out by distinct frontal and posterior cortical networks, but the degree and timing of their recruitment and dynamics remain unknown. We present a novel deep learning architecture that translates neural signals recorded directly from cortex to an interpretable representational space that can reconstruct speech. We leverage state-of-the-art learnt decoding networks to disentangle feedforward vs. feedback processing. Unlike prevailing models, we find a mixed cortical architecture in which frontal and temporal networks each process both feedforward and feedback information in tandem. We elucidate the timing of feedforward and feedback related processing by quantifying the derived receptive fields. Our approach provides evidence for a surprisingly mixed cortical architecture of speech circuitry together with decoding advances that have important implications for neural prosthetics.

neuroscience↗

Moment-by-moment tracking of naturalistic learning and its underlying hippocampo-cortical interactions

Every day our memory system achieves a remarkable feat: We form lasting memories of stimuli that were only encountered once. Here we investigate such learning as it naturally occurs during story listening, with the goal of uncovering when and how memories are stored and retrieved during processing of continuous, naturalistic stimuli. In behavioral experiments we confirm that, after a single exposure to a naturalistic story, participants can learn about its structure and are able to recall upcoming words in the story. In patients undergoing electrocorticographic recordings, we then track mnemonic information in high frequency activity (70 - 200Hz) as patients listen to a story twice. In auditory processing regions we demonstrate the rapid reinstatement of upcoming information after a single exposure; this neural measure of predictive recall correlates with behavioral measures of event segmentation and learning. Connectivity analyses on the neural data reveal information-flow from cortex to hippocampus at the end of events. On the second time of listening information-flow from hippocampus to cortex precedes moments of successful reinstatement.

neuroscience↗

Thinking ahead: prediction in context as a keystone of language in humans and machines

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models are trained to generate appropriate linguistic responses in a given context. We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process natural language: 1) both are engaged in continuous next-word prediction before word-onset; 2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise (i.e., prediction error signals); 3) both represent words as a function of the previous context. In support of these three principles, our findings indicate that: a) the neural activity before word-onset contains context-dependent predictive information about forthcoming words, even hundreds of milliseconds before the words are perceived; b) the neural activity after word-onset reflects the surprise level and prediction error; and c) autoregressive DLM contextual embeddings capture the neural representation of context-specific word meaning better than arbitrary or static semantic embeddings. Together, our findings suggest that autoregressive DLMs provide a novel and biologically feasible computational framework for studying the neural basis of language.

neuroscience↗

Proprioception Has Limited Influence on Tactile Reference Frame Selection

Perceiving the spatial location and physical dimensions of objects that we touch is crucial for goal-directed actions. To achieve this, our brain transforms skin-based coordinates into a reference frame by integrating visual and proprioceptive cues, a process known as tactile remapping. In the current study, we examine the role of proprioception in the remapping process when information from the more dominant visual modality is withheld. We developed a new visual-to-touch sensory substitution device and asked participants to perform a spatial localization task in three different arm postures that included posture switches between blocks of trials. We observed that in the absence of visual information novel proprioceptive inputs can be overridden after switching postures. This behavior demonstrates effective top-down modulations of proprioception and points to the unequal contribution of different sensory modalities to tactile remapping.

neuroscience↗

Proteomic and Transcriptomic Analyses of the Hippocampus and Cortex in SUDEP and High-Risk SUDEP Cases

Sudden unexpected death in epilepsy (SUDEP) is the leading type of epilepsy-related death. Severely depressed brain activity in these cases may impair respiration, arousal, and protective reflexes, occurring as a prolonged postictal generalized EEG suppression (PGES) and resulting in a high-risk for SUDEP. In autopsy hippocampus and cortex, we observed no proteomic differences between SUDEP and epilepsy cases, contrasting our previously reported robust differences between epilepsy and controls. Transcriptomics in hippocampus and cortex from surgical epilepsy cases segregated by PGES identified 55 differentially expressed genes (37 protein-coding, 15 lncRNAs, three pending) in hippocampus. Overall, the SUDEP proteome and high-risk SUDEP transcriptome largely reflected other epilepsy cases in the brain regions analyzed, consistent with diverse epilepsy syndromes and comorbidities associated with SUDEP. Thus, studies with larger cohorts and different epilepsy syndromes, as well as additional anatomic regions may identify molecular mechanisms of SUDEP.

neuroscience↗