Biology subjects

Pitkow, X.

Publications and source records attributed to Pitkow, X..

At least 19 recordsLinked to original sources

bioRxiv2026

Intelligent systems distribute memory across internal and embodied resources

Intelligent systems do not operate in isolation from the body, yet how bodily actions support internal computation remains unclear. Here we show that primates and artificial agents recruit eye movements to maintain beliefs about hidden goals when internal computation is constrained. In a naturalistic navigation task, both systems spontaneously tracked goals with their eyes after they disappeared from view. This strategy emerged in artificial agents only when internal memory was constrained, identifying resource constraints as a driver of memory externalization. Restricting gaze impaired performance in both systems, while natural behavioral variability and causal perturbations in agents established a direct influence of oculomotor state on internal beliefs about hidden goals. Across agents, varying the reliability of internal memory and gaze redistributed goal information between internal and embodied states. The same resource-allocation principle predicted the distribution of goal information in macaque parietal cortex, where greater reliance on gaze was accompanied by weaker gaze-independent neural representations. Together, these findings identify eye movements as embodied memory and reveal a convergent principle across intelligent systems: memory is allocated across internal and embodied resources according to their reliability.

neuroscience↗

bioRxiv2026

Rationalizing sensorimotor deficits in autism spectrum disorder

Autism spectrum disorder (ASD) is a neurodevelopmental condition characterized by socio-communicative, sensory, and motor anomalies. Recently, a host of probabilistic models have been proposed to account for the varied phenotypes observed in ASD. These computational psychiatry efforts, however, have (1) largely focused on simple tasks that artificially segregate action from perception, and (2) unequivocally assumed that both neurotypical and divergent populations attempt to optimize the same cost function. Here, we ask individuals with ASD and neurotypical (NT) controls to perform a task in virtual reality (VR): they should navigate to, and stop at, the position of briefly presented targets in a closed action-perception loop. First, we demonstrate that while summary task performance is insufficient to distinguish between ASD and NT individuals, data-rich virtual trajectories do differentiate these groups. This suggests that ASD and control participants use distinct navigational strategies and/or cost functions. Next, we leverage the theoretical framework of Inverse Rational Control (IRC) to interpret the sensorimotor anomalies present in ASD. IRC does not assume that agents are optimal, but rational: that is, agents attempt to maximize reward according to an internal model of the task, which may not match the true task and environment. Applying IRC to this VR navigation task revealed both sensory and motor differences between ASD and neurotypical participants. The sensory differences are consistent with Bayesian inference under altered weighting of sensory evidence. The motor differences manifest as more frequent and abrupt actions in ASD, which we explain by a reduced subjective action cost in ASD. Together, these results demonstrate the utility of model-based characterization of psychopathology and suggest that the computational study of ASD should consider not only perceptual anomalies, but also subjective costs.

neuroscience↗

bioRxiv2026

Visual uncertainty and task demands shape active sensing strategies in mice

In natural environments, animals actively sample visual information to guide behavior. Sensory feedback is dynamic and often requires active movements, whether saccading across the lines of this page or walking through a park. From high-acuity vision in hawks to low-acuity mice, many animals actively navigate to seek information, which can be called infotaxis. Although mice have relatively low-acuity vision, they still rely on sight for critical behaviors including navigation and prey capture. Yet, how sensitive they are to visual information and performing infotaxis has not been established. Here, we develop a virtual reality object discrimination task to investigate visual decision-making under naturalistic conditions. We show that mice perform infotaxis by actively seeking out informative views to guide their choices. Stimulus manipulations confirm that this strategy is modulated by the amount of available visual information. These results reveal that mice use principled active strategies to resolve visual uncertainty, highlighting a key role for information-seeking in natural vision. HighlightsO_LIMice perform active visual sensing in a free-range VR object discrimination task. C_LIO_LIThey perform infotaxis and select the correct object under variable visibility. C_LIO_LIDecisions decoded from head-body movements and speed reflect occlusion sensitivity. C_LI

neuroscience↗

bioRxiv2026

Temporal Structure of Reward Availability and Sensory Uncertainty Modulate Allocation Dynamics in Naturalistic Foraging

Adaptive foraging requires animals to combine uncertain sensory cues with predictions about when rewards are likely to occur. While theoretical models describe how animals should allocate their effort under variable-interval reward schedules, it remains unclear how the timing or rewards and the reliability of sensory cues affects behavior. We developed a continuous foraging task in which freely moving macaque monkeys navigated among three reward patches. Rewards became available at unpredictable times, with their availability signaled by a visual cue of varying reliability. We also varied the schedule of reward availability: in some conditions, rewards were equally likely to become available at any moment (exponentially distributed intervals), while in others the interval distribution was more concentrated around a particular mean (gamma-distributed intervals) which increased the cost of premature responses. Under exponential schedules, monkeys eventually allocated their time at different patches according to reward schedules, and cue reliability had only modest effects. Under gamma-distributed intervals, monkeys more quickly learned to differentiate between patches. Their choices were more strongly dependent on predicted reward timing, particularly when sensory cues were highly reliable. These results show that both the timing of rewards and reliability of sensory cues shape how animals allocate their time and effort in continuous naturalistic foraging tasks.

animal behavior and cognition↗

bioRxiv2025

The neural basis of emotional generalization in empathy

The essence of empathy is generalization of emotion across persons. Here, we leverage recent theoretical advances in the neuroscience of generalization to help us understand empathy. We measured brain activity in human neurosurgical patients performing two tasks, one focused on identifying their own emotional response and one identifying emotional responses in others. We quantified the representational geometry of local field potential (LFP) high-gamma activity in four regions: the medial temporal lobe, anterior cingulate cortex, orbitofrontal cortex, and insula. We found encoding of both self- and other-emotions in all four regions, but codes for emotion and person are disentangled (that is, factorized) in the insula, but not the other regions. This factorized representation allows for cross-person generalization of emotion in a way that tangled (non-factorized) representations do not. Together, these results support the hypothesis that the insula uniquely contributes to social mirroring processes by which we understand emotions across individuals.

neuroscience↗

bioRxiv2025

Low-Rank Tensor Encoding Models Decompose Natural Speech Comprehension Processes

How does the brain process language over time? Research suggests that natural human language is processed hierarchically across brain regions over time. However, attempts to characterize this computation have thus far been limited to tightly controlled experimental settings that capture only a coarse picture of the brain dynamics underlying human natural language comprehension. The recent emergence of LLM encoding models promises a new avenue to discover and characterize rich semantic information in the brain, yet interpretable methods for linking information in LLMs to language processing over time are limited. In this work, we develop a low-rank tensor regression method to decompose LLM encoding models into interpretable components of semantics, time, and brain region activation, and apply the method to a Magnetoencephalography (MEG) dataset in which subjects listened to narrative stories. With only a few components, we show improved performance compared to a standard ridge regression encoding model, suggesting the low-rank models provide a good inductive bias for language encoding. In addition, our method discovers a diverse spectrum of interpretable response components that are sensitive to a rich set of low-level and semantic language features, showing that our method is able to separate distinct language processing features in neural signals. After controlling for low-level audio and sentence features, we demonstrate better capture of semantic features. Through use of low-rank tensor encoding models we are able to decompose neural responses to language features, showing improved encoding performance and interpretable processing components, suggesting our method as a useful tool for uncovering language processes in naturalistic settings.

neuroscience↗

bioRxiv2024

Dorsolateral prefrontal cortex drives strategic aborting by optimizing long-run policy extraction

Real world choices often involve balancing decisions that are optimized for the short-vs. long-term. Here, we reason that apparently sub-optimal single trial decisions in macaques may in fact reflect long-term, strategic planning. We demonstrate that macaques freely navigating in VR for sequentially presented targets will strategically abort offers, forgoing more immediate rewards on individual trials to maximize session-long returns. This behavior is highly specific to the individual, demonstrating that macaques reason about their own long-run performance. Reinforcement-learning (RL) models suggest this behavior is algorithmically supported by modular actor-critic networks with a policy module not only optimizing long-term value functions, but also informed of specific state-action values allowing for rapid policy optimization. The behavior of artificial networks suggests that changes in policy for a matched offer ought to be evident as soon as offers are made, even if the aborting behavior occurs much later. We confirm this prediction by demonstrating that single units and population dynamics in macaque dorsolateral prefrontal cortex (dlPFC), but not parietal area 7a or dorsomedial superior temporal area (MSTd), reflect the upcoming reward-maximizing aborting behavior upon offer presentation. These results cast dlPFC as a specialized policy module, and stand in contrast to recent work demonstrating the distributed and recurrent nature of belief-networks.

neuroscience↗

bioRxiv2024

Insula uses overlapping codes for emotion in self and others

In daily life, we must recognize others emotions so we can respond appropriately. This ability may rely, at least in part, on neural responses similar to those associated with our own emotions. We hypothesized that the insula, a cortical region near the junction of the temporal, parietal, and frontal lobes, may play a key role in this process. We recorded local field potential (LFP) activity in human neurosurgical patients performing two tasks, one focused on identifying their own emotional response and one on identifying facial emotional responses in others. We found matching patterns of gamma- and high-gamma band activity for the two tasks in the insula. Three other regions (MTL, ACC, and OFC) clearly encoded both self- and other-emotions, but used orthogonal activity patterns to do so. These results support the hypothesis that the insula plays a particularly important role in mediating between experienced vs. observed emotions.

neuroscience↗

bioRxiv2023

Towards a Foundation Model of the Mouse Visual Cortex

The complexity of neural circuits makes it challenging to decipher the brains algorithms of intelligence. Recent break-throughs in deep learning have produced models that accurately simulate brain activity, enhancing our understanding of the brains computational objectives and neural coding. However, these models struggle to generalize beyond their training distribution, limiting their utility. The emergence of foundation models, trained on vast datasets, has introduced a new AI paradigm with remarkable generalization capabilities. We collected large amounts of neural activity from visual cortices of multiple mice and trained a foundation model to accurately predict neuronal responses to arbitrary natural videos. This model generalized to new mice with minimal training and successfully predicted responses across various new stimulus domains, such as coherent motion and noise patterns. It could also be adapted to new tasks beyond neural prediction, accurately predicting anatomical cell types, dendritic features, and neuronal connectivity within the MICrONS functional connectomics dataset. Our work is a crucial step toward building foundation brain models. As neuroscience accumulates larger, multi-modal datasets, foundation models will uncover statistical regularities, enabling rapid adaptation to new tasks and accelerating research.

neuroscience↗

bioRxiv2023

Bipartite invariance in mouse primary visual cortex

A primary goal of sensory systems is to extract robust and meaningful features that are invariant to variations in the sensory input. Characterizing these invariances at the neuronal level is crucial for understanding how the visual system supports generalization, but the high-dimensional nature of ecological stimuli poses major challenges. Consequently, our understanding of how the brain represents invariances has historically depended on a few examples, such as phase invariance to grating stimuli in V1 complex cells. Here, we leverage the inception loop paradigm --iterating between large-scale recordings, deep learning neuronal predictive models, and in silico experiments with in vivo verification--to characterize neuronal invariances in mouse V1. Using a neuronal predictive model, we synthesized Diverse Exciting Inputs (DEIs) that strongly drive target neurons while differing substantially in image space. These DEIs revealed a novel bipartite invariance: one portion of the receptive field encodes shift-invariant, high-frequency textures, while the other encodes a fixed, low-frequency spatial pattern. This subfield division aligned with object boundaries defined by spatial frequency differences in highly activating stimuli, suggesting bi-partite invariance contributes to segmentation. Our analysis of computational models and anatomical data from the MICrONS dataset revealed a hierarchical organization of excitatory neurons in mouse V1 Layers 2/3: We found that postsynaptic neurons exhibited greater invariance than their presynaptic inputs, while neurons with lower invariance formed more connections. These findings suggest a synaptic-level hierarchy that progressively increases neural invariance within the primary visual cortex. Intriguingly, similar high-low frequency bipartite patterns strongly activate certain units in artificial neural networks, suggesting that universal visual representations govern both biological and artificial systems, potentially aiding in the extraction of visual features from complex backgrounds.

neuroscience↗

bioRxiv2023

Pattern completion and disruption characterize contextual modulation in mouse visual cortex

Vision is fundamentally context-dependent, with neuronal responses influenced not just by local features but also by surrounding contextual information. In the visual cortex, studies using simple grating stimuli indicate that congruent stimuli--where the center and surround share the same orientation--are more inhibitory than when orientations are orthogonal, potentially serving redundancy reduction and predictive coding. Understanding these center-surround interactions in relation to natural image statistics is challenging due to the high dimensionality of the stimulus space, yet crucial for deciphering the neuronal code of real-world sensory processing. Utilizing large-scale recordings from mouse V1, we trained convolutional neural networks (CNNs) to predict and synthesize surround patterns that either optimally suppressed or enhanced responses to center stimuli, confirmed by in v ivo experiments. Contrary to the notion that congruent stimuli are suppressive, we found that surrounds that completed patterns based on natural image statistics were facilitatory, while disruptive surrounds were suppressive. Applying our CNN image synthesis method in macaque V1, we discovered that pattern completion within the near surround occurred more frequently with excitatory than with inhibitory surrounds, suggesting that our results in mice are conserved in macaques. Further, experiments and model analyses confirmed previous studies reporting the opposite effect with grating stimuli in both species. Using the MICrONS functional connectomics dataset, we observed that neurons with similar feature selectivity formed excitatory connections regardless of their receptive field overlap, aligning with the pattern completion phenomenon observed for excitatory surrounds. Finally, our empirical results emerged in a normative model of perception implementing Bayesian inference, where neuronal responses are modulated by prior knowledge of natural scene statistics. In summary, our findings identify a novel relationship between contextual information and natural scene statistics and provide evidence for a role of contextual modulation in hierarchical inference.

neuroscience↗

bioRxiv2023

Cell-type-specific inhibitory circuitry from a connectomic census of mouse visual cortex

Mammalian cortex features a vast diversity of neuronal cell types, each with characteristic anatomical, molecular and functional properties. Synaptic connectivity powerfully shapes how each cell type participates in the cortical circuit, but mapping connectivity rules at the resolution of distinct cell types remains difficult. Here, we used millimeter-scale volumetric electron microscopy1 to investigate the connectivity of all inhibitory neurons across a densely-segmented neuronal population of 1352 cells spanning all layers of mouse visual cortex, producing a wiring diagram of inhibitory connections with more than 70,000 synapses. Taking a data-driven approach inspired by classical neuroanatomy, we classified inhibitory neurons based on the relative targeting of dendritic compartments and other inhibitory cells and developed a novel classification of excitatory neurons based on the morphological and synaptic input properties. The synaptic connectivity between inhibitory cells revealed a novel class of disinhibitory specialist targeting basket cells, in addition to familiar subclasses. Analysis of the inhibitory connectivity onto excitatory neurons found widespread specificity, with many interneurons exhibiting differential targeting of certain subpopulations spatially intermingled with other potential targets. Inhibitory targeting was organized into "motif groups," diverse sets of cells that collectively target both perisomatic and dendritic compartments of the same excitatory targets. Collectively, our analysis identified new organizing principles for cortical inhibition and will serve as a foundation for linking modern multimodal neuronal atlases with the cortical wiring diagram.

neuroscience↗

bioRxiv2022

Large-scale unsupervised discovery of excitatory morphological cell types in mouse visual cortex

Neurons in the neocortex exhibit astonishing morphological diversity which is critical for properly wiring neural circuits and giving neurons their functional properties. However, the organizational principles underlying this morphological diversity remain an open question. Here, we took a data-driven approach using graph-based machine learning methods to obtain a low-dimensional morphological "bar code" describing more than 30,000 excitatory neurons in mouse visual areas V1, AL and RL that were reconstructed from the millimeter scale MICrONS serial-section electron microscopy volume. Contrary to previous classifications into discrete morphological types (m-types), our data-driven approach suggests that the morphological landscape of cortical excitatory neurons is better described as a continuum, with a few notable exceptions in layers 5 and 6. Dendritic morphologies in layers 2-3 exhibited a trend towards a decreasing width of the dendritic arbor and a smaller tuft with increasing cortical depth. Inter-area differences were most evident in layer 4, where V1 contained more atufted neurons than higher visual areas. Moreover, we discovered neurons in V1 on the border to layer 5 which avoided deeper layers with their dendrites. In summary, we suggest that excitatory neurons morphological diversity is better understood by considering axes of variation than using distinct m-types.

neuroscience↗

bioRxiv2022

Inductive biases of neural networks for generalization in spatial navigation

The brain may have evolved a modular architecture for reward-based learning in daily tasks, with circuits featuring functionally specialized modules that match the task structure. We propose that this architecture enables better learning and generalization than architectures with less specialized modules. To test this hypothesis, we trained reinforcement learning agents with various neural architectures on a naturalistic navigation task. We found that the architecture that largely segregates computations of state representation, value, and action into specialized modules enables more efficient learning and better generalization. Behaviors of agents with this architecture also resemble macaque behaviors more closely. Investigating the latent state computations in these agents, we discovered that the learned state representation combines prediction and observation, weighted by their relative uncertainty, akin to a Kalman filter. These results shed light on the possible rationale for the brains modular specializations and suggest that artificial systems can use this insight from neuroscience to improve learning and generalization in natural tasks.

neuroscience↗

bioRxiv2022

Hierarchical Modular Structure of the Drosophila Connectome

The structure of neural circuitry plays a crucial role in brain function. Previous studies of brain organization generally had to trade off between coarse descriptions at a large scale and fine descriptions on a small scale. Researchers have now reconstructed tens to hundreds of thousands of neurons at synaptic resolution, enabling investigations into the interplay between global, modular organization, and cell type-specific wiring. Analyzing data of this scale, however, presents unique challenges. To address this problem we applied novel community detection methods to analyze the synapse-level reconstruction of an adult fruit fly brain containing over 20 thousand neurons and 10 million synapses. Using a machine-learning algorithm, we find the most densely connected communities of neurons by maximizing a generalized modularity density measure. We resolve the community structure at a range of scales, from large (on the order of thousands of neurons) to small (on the order of tens of neurons). We find that the network is organized hierarchically and larger-scale communities are composed of smaller-scale structures. Our methods identify well-known features of the fly brain, including its sensory pathways. Moreover, focusing on specific brain regions, we are able to identify subnetworks with distinct connectivity types. For example, manual efforts have identified layered structures in the fan-shaped body. Our methods not only automatically recover this layered structure, but also resolve finer connectivity patterns to downstream and upstream areas. We also find a novel modular organization of the superior neuropil, with distinct clusters of upstream and downstream brain regions dividing the neuropil into several pathways. These methods show that the fine-scale, local network reconstruction made possible by modern experimental methods are sufficiently detailed to identify the organization of the brain across scales, and enable novel predictions about the structure and function of its parts.

neuroscience↗

bioRxiv2022

Quantitative Census of Local Somatic Features in Mouse Visual Cortex

Mammalian neocortex contains a highly diverse set of cell types. These types have been mapped systematically using a variety of molecular, electrophysiological and morphological approaches. Each modality offers new perspectives on the variation of biological processes underlying cell type specialization. Cellular scale electron microscopy (EM) provides dense ultrastructural examination and an unbiased perspective into the subcellular organization of brain cells, including their synaptic connectivity and nanometer scale morphology. It also presents a clear challenge for analysis to identify cell-types in data that contains tens of thousands of neurons, most of which have incomplete reconstructions. To address this challenge, we present the first systematic survey of the somatic region of all cells within a cubic millimeter of cortex using quantitative features obtained from EM. This analysis demonstrates a surprising sufficiency of the perisomatic region to identify cell-types, including types defined primarily based on their connectivity patterns. We then describe how this classification facilitates cell type specific connectivity characterization and locating cells with rare connectivity patterns in the dataset.

neuroscience↗

bioRxiv2022

Robust deep learning object recognition models rely on low frequency information in natural images

Machine learning models have difficulty generalizing to data outside of the distribution they were trained on. In particular, vision models are usually vulnerable to adversarial attacks or common corruptions, to which the human visual system is robust. Recent studies have found that regularizing machine learning models to favor brain-like representations can improve model robustness, but it is unclear why. We hypothesize that the increased model robustness is partly due to the low spatial frequency preference inherited from the neural representation. We tested this simple hypothesis with several frequency-oriented analyses, including the design and use of hybrid images to probe model frequency sensitivity directly. We also examined many other publicly available robust models that were trained on adversarial images or with data augmentation, and found that all these robust models showed a greater preference to low spatial frequency information. We show that preprocessing by blurring can serve as a defense mechanism against both adversarial attacks and common corruptions, further confirming our hypothesis and demonstrating the utility of low spatial frequency information in robust object recognition.

neuroscience↗

bioRxiv2022

Dynamical Latent State Computation in the Posterior Parietal Cortex

Success in many real-world tasks depends on our ability to dynamically track hidden states of the world. To understand the underlying neural computations, we recorded brain activity in posterior parietal cortex (PPC) of monkeys navigating by optic flow to a hidden target location within a virtual environment, without explicit position cues. In addition to sequential neural dynamics and strong interneuronal interactions, we found that the hidden state - monkeys displacement from the goal - was encoded in single neurons, and could be dynamically decoded from population activity. The decoded estimates predicted navigation performance on individual trials. Task manipulations that perturbed the world model induced substantial changes in neural interactions, and modified the neural representation of the hidden state, while representations of sensory and motor variables remained stable. The findings were recapitulated by a task-optimized recurrent neural network model, suggesting that neural interactions in PPC embody the world model to consolidate information and track task-relevant hidden states.

neuroscience↗