Biology subjects

Naselaris, T.

Publications and source records attributed to Naselaris, T..

15 recordsLinked to original sources

bioRxiv2025

A transformation from vision to imagery in the human brain

Extensive work has shown that the visual cortex is reactivated during mental imagery, and that models trained on visual data can predict imagery activity and decode imagined stimuli. These findings may explain why imagery can feel and function like vision, but give little insight into how the brain activity patterns that encode seen and mental images differ. While one popular theory ("weak vision") suggests that imagery differs from vision only in strength, recent work points to more complex differences. To clarify the relationship between visual and imagery activity in the human brain, we introduce the concept of an imagery transformation--a mapping from visual to imagery activity patterns evoked by the same stimulus. Importantly, this approach can describe a variety of possible scenarios, from simple rescaling to the selective removal or reorientation of activity dimensions. Using two 7T fMRI datasets, we estimated imagery transformations across different visual areas and found they accurately predict imagery activity. We show that imagery transformations are indeed more complex than simple weakening: in early visual cortex, they halve the number of active dimensions and reorient them, such that reconstructions of visual activity in terms of imagery dimensions explain only 25-50% of the variance. Nonetheless, these reoriented dimensions still coarsely encode the features encoded by the principal dimensions of visual activity. These findings help to explain the "same but different" relationship between mental imagery and vision: imagery activity patterns are transformations of visual activity patterns that approximate them, but encode fewer features and occupy a distinct subspace within the overall space of brain activations.

neuroscience↗

bioRxiv2024

Variation in the geometry of concept manifolds acrosshuman visual cortex

Brain activity patterns in high-level visual cortex support accurate linear classification of visual concepts (e.g., objects or scenes). It has long been appreciated that the accuracy of linear classification in any brain area depends on the geometry of its concept manifolds--sets of brain activity patterns that encode images of a concept. However, it is unclear how the geometry of concept manifolds differs between regions of visual cortex that support accurate classification and those that dont, or how it differs between visual cortex and deep neural networks (DNNs). We estimated geometric properties of concept manifolds that, per a recent theory, directly determine the accuracy of simple "few-shot" linear classifiers. Using a large fMRI dataset, we show that variation in classification accuracy across human visual cortex is driven by a variation in a single geometric property: the distance between manifold centers ("geometric Signal"). In contrast, variation in classification accuracy across most DNN layers is driven by an increase in the effective number of manifold dimensions ("Dimensionality"). Despite this difference in the geometric properties that affect few-shot classification performance in the brain and DNNs, we find that Signal and Dimensionality are strongly, negatively correlated: when Signal increases across brain regions or DNN layers, Dimensionality decreases, and vice versa. We conclude that visual cortex and DNNs deploy different geometric strategies for accurate linear classification of concepts, even though both are subject to the same constraint.

neuroscience↗

bioRxiv2024

Vicarious Somatotopic Maps Tile Visual Cortex

Our sensory systems work together to generate a cohesive experience of the world around us. Watching others being touched activates brain areas representing our own sense of touch: the visual system recruits touch-related computations to simulate bodily consequences of visual inputs. One long-standing question is how the brain implements this interface between visual and somatosensory representations. To address this question, we developed a method to simultaneously map somatosensory body part tuning and visual field tuning throughout the brain. Applying this method on ongoing co-activations during rest resulted in detailed maps of the body-part tuning in the brains endogenous somatotopic network. During movie watching, somatotopic tuning explains responses throughout the entire dorsolateral visual system, revealing an array of somatotopic body maps that tile the cortical surface. The tuning of these maps aligned with those of visual maps, and predicted both preferences for visual field locations and the visual-category preferences for body parts. These results reveal a mode of brain organization in which aligned visual-somatosensory topographic maps connect visual and bodily reference frames. This cross-modal interface is ideally situated to translate raw sensory impressions into more abstract formats useful for action, social cognition, and semantic processing.

neuroscience↗

bioRxiv2024

Temporal asymmetry of neural representations predicts memory decisions

A stimulus can be familiar for multiple reasons. It might have been recently encountered, or is similar to recent experience, or is similar to typical experience. Understanding how the brain translates these sources of similarity into memory decisions is a fundamental, but challenging goal. Here, using fMRI, we computed neural similarity between a current stimulus and events from different temporal windows in the past and future (from seconds to days). We show that trial-by-trial memory decisions (is this stimulus old?) were predicted by the difference in similarity to past vs. future events (temporal asymmetry). This relationship was (i) evident in lateral parietal and occipitotemporal cortices, (ii) strongest when considering events from the recent past (minutes ago), and (iii) most pronounced when veridical (true) memories were weak. These findings suggest a new perspective in which the brain supports memory decisions by comparing what actually occurred to what is likely to occur.

neuroscience↗

bioRxiv2024

Disentangling signal and noise in neural responses through generative modeling

Measurements of neural responses to identically repeated experimental events often exhibit large amounts of variability. This noise is distinct from signal, operationally defined as the average expected response across repeated trials for each given event. Accurately distinguishing signal from noise is important, as each is a target that is worthy of study (many believe noise reflects important aspects of brain function) and it is important not to confuse one for the other. Here, we describe a principled modeling approach in which response measurements are explicitly modeled as the sum of samples from multivariate signal and noise distributions. In our proposed method--termed Generative Modeling of Signal and Noise (GSN)--the signal distribution is estimated by subtracting the estimated noise distribution from the estimated data distribution. Importantly, GSN improves estimates of the signal distribution, but does not provide improved estimates of responses to individual events. We validate GSN using ground-truth simulations and show that it compares favorably with related methods. We also demonstrate the application of GSN to empirical fMRI data to illustrate a simple consequence of GSN: by disentangling signal and noise components in neural responses, GSN denoises principal components analysis and improves estimates of dimensionality. We end by discussing other situations that may benefit from GSNs characterization of signal and noise, such as estimation of noise ceilings for computational models of neural activity. A code toolbox for GSN is provided with both MATLAB and Python implementations.

neuroscience↗

bioRxiv2023

Stacked regressions and structured variance partitioning for interpretable brain maps

Relating brain activity associated with a complex stimulus to different properties of that stimulus is a powerful approach for constructing functional brain maps. However, when stimuli are naturalistic, their properties are often correlated (e.g., visual and semantic features of natural images, or different layers of a convolutional neural network that are used as features of images). Correlated properties can act as confounders for each other and complicate the interpretability of brain maps, and can impact the robustness of statistical estimators. Here, we present an approach for brain mapping based on two proposed methods: stacking different encoding models and structured variance partitioning. Our stacking algorithm combines encoding models that each use as input a feature space that describes a different stimulus attribute. The algorithm learns to predict the activity of a voxel as a linear combination of the outputs of different encoding models. We show that the resulting combined model can predict held-out brain activity better or at least as well as the individual encoding models. Further, the weights of the linear combination are readily interpretable; they show the importance of each feature space for predicting a voxel. We then build on our stacking models to introduce structured variance partitioning, a new type of variance partitioning that takes into account the known relationships between features. Our approach constrains the size of the hypothesis space and allows us to ask targeted questions about the similarity between feature spaces and brain regions even in the presence of correlations between the feature spaces. We validate our approach in simulation, showcase its brain mapping potential on fMRI data, and release a Python package. Our methods can be useful for researchers interested in aligning brain activity with different layers of a neural network, or with other types of correlated feature spaces.

neuroscience↗

bioRxiv2022

Incorporating natural language into vision models improves prediction and understanding of higher visual cortex

Advances in neural networks have been catalyzed by joint training on images and natural language, increased dataset sizes, and data diversity. We explored whether the same factors support similar improvements in predicting visual responses in the human brain. We used models pre-trained with Contrastive Language-Image Pre-training (CLIP) - which learns image embeddings that best match text embeddings of image captions from diverse, large-scale datasets - to study visual representations. We built voxelwise encoding models based on CLIP image features to predict brain responses to real-world images. ResNet50 with CLIP explained up to R2 = 79% of variance in individual voxel responses in held-out test data, a significant increase from models trained only with image/label pairs (ImageNet trained ResNet) or text (BERT). Comparisons across different model backbones ruled out network architecture as a factor in performance improvements. Comparisons across models that controlled for dataset size and data diversity demonstrated that language feedback along with data diversity in larger datasets are important factors in explaining neural responses in high-level visual brain regions. Visualizations of model embeddings and Principal Component Analysis (PCA) revealed that our models capture both global and fine-grained semantic dimensions represented within human visual cortex.

neuroscience↗

bioRxiv2022

Re-expression of CA1 and entorhinal activity patterns preserves temporal context memory at long timescales

Converging, cross-species evidence indicates that memory for time is supported by hippocampal area CA1 and entorhinal cortex. However, limited evidence characterizes how these regions preserve temporal memories over long timescales (e.g., months). At long timescales, memoranda may be encountered in multiple temporal contexts, potentially creating interference. Here, using 7T fMRI, we measured CA1 and entorhinal activity patterns as human participants viewed thousands of natural scene images distributed, and repeated, across many months. We show that memory for an images original temporal context was predicted by the degree to which CA1/entorhinal activity patterns from the first encounter with an image were re-expressed during re-encounters occurring minutes to months later. Critically, temporal memory signals were dissociable from predictors of recognition confidence, which were carried by distinct medial temporal lobe expressions. These findings suggest that CA1 and entorhinal cortex preserve temporal memories across long timescales by coding for and reinstating temporal context information.

neuroscience↗

bioRxiv2022

Color-biased regions in the ventral visual pathway are food-selective

The ventral visual pathway is well known to be involved in recognizing and categorizing objects (Kanwisher and Dilks, 2013). Three color-biased areas have also been found between face and place selective areas in the ventral visual pathway (Lafer-Sousa et al., 2016). To understand the function of these color-biased areas in a region known for object recognition, we analyzed the Natural Scenes Dataset (NSD; Allen et al., 2022), a large 7T fMRI dataset from 8 participants who viewed up to 30,000 trials of images of colored natural scenes. In a whole-brain analysis, we correlated the average color saturation of the images and the voxel responses, revealing color-biased areas that diverge into two streams in the ventral visual pathway, beginning in V4 and extending medially and laterally of the Fusiform Face Area in both hemispheres. We drew regions of interest (ROIs) for the two streams and found that the images for each ROI that evoked the largest responses had certain characteristics: They contained food, contained circular objects, had higher color saturation, contained warmer hues, and had more luminance entropy. A multiple linear regression showed that the presence of food in images was the strongest predictor of voxel responses in the medial and lateral color-biased regions for all eight participants, but that color saturation also contributed independently to voxel responses. Our results show that these areas are food-selective and color biased. We suggest that these streams might be involved in using color to recognize and judge the properties of food.

neuroscience↗

bioRxiv2022

Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex

Deep neural networks (DNNs) trained to perform visual tasks learn representations that align with the hierarchy of visual areas in the primate brain. This finding has been taken to imply that the primate visual system forms representations by passing them through a hierarchical sequence of brain areas, just as DNNs form representations by passing them through a hierarchical sequence of layers. To test the validity of this assumption, we optimized DNNs not to perform visual tasks but to directly predict brain activity in human visual areas V1-V4. Using a massive sampling of human brain activity, we constructed brain-optimized networks that predict brain activity even more accurately than task-optimized networks. We show that brain-optimized networks can learn representations that diverge from those formed in a strict hierarchy. Brain-optimized networks do not need to align representations in V1-V4 with layer depth; moreover, they are able to accurately model anterior brain areas (e.g., V4) without computing intermediary representations associated with posterior brain areas (e.g., V1). Our results challenge the view that human visual areas V1-V4 act--like the early layers of a DNN--as a serial pre-processing sequence for higher areas, and suggest they may subserve their own independent functions.

neuroscience↗

bioRxiv2021

A massive 7T fMRI dataset to bridge cognitive and computational neuroscience

Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. We present the Natural Scenes Dataset (NSD), in which high-resolution fMRI responses to tens of thousands of richly annotated natural scenes are measured while participants perform a continuous recognition task. To optimize data quality, we develop and apply novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we use NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality, and breadth, NSD opens new avenues of inquiry in cognitive and computational neuroscience.

neuroscience↗

bioRxiv2019

Voxel-to-voxel predictive models reveal unexpected structure in unexplained variance

Encoding models based on deep convolutional neural networks (DCNN) more accurately predict BOLD responses to natural scenes in the visual system than any other currently available model. However, DCNN-based encoding models fail to predict a significant amount of variance in the activity of most voxels in all visual areas. This failure could reflect limitations in the data (e.g., a noise ceiling), or could reflect limitations of the DCNN as a model of computation in the brain. Understanding the source and structure of the unexplained variance could therefore provide helpful clues for improving models of brain computation. Here, we characterize the structure of the variance that DCNN-based encoding models cannot explain. Using a publicly available dataset of BOLD responses to natural scenes, we determined if the source of unexplained variance was shared across voxels, individual brains, retinotopic locations, and hierarchically distant visual brain areas. We answered these questions using voxel-to-voxel (vox2vox) models that predict activity in a target voxel given activity in a population of source voxels. We found that simple linear vox2vox models increased within-subject prediction accuracy over DCNN-based models for any pair of source/target visual areas, clearly demonstrating that the source of unexplained variance is widely shared within and across visual brain areas. However, vox2vox models were not more accurate than DCNN-based models when source and target voxels came from separate brains, demonstrating that the source of unexplained variance was not shared across brains. Furthermore, the weights of these vox2vox models permitted explicit readout of the receptive field location of target voxels, demonstrating that the source of unexplained variance induces correlations primarily between the activities of voxels with overlapping receptive fields. Finally, we found that vox2vox model prediction accuracy was heavily dependent upon the signed hierarchical distance between the source and target voxels: for feed-forward models (source area lower in the visual hierarchy than target area) prediction accuracy decreased with hierarchical distance between source and target. It did not decrease for feedback models. In contrast, the same analysis applied across layers of a DCNN did not reveal this feed-forward/feedback asymmetry. Given these results, we argue that the structured variance unexplained by DCNN-based encoding models is unlikely to be entirely caused by spatially correlated noise or eye movements; rather, our results point to a need for brain models that include endogenous dynamics and a pattern of connectivity that is not strictly feed-forward.

neuroscience↗

bioRxiv2018

Human brain activity during mental imagery exhibits signatures of inference in a hierarchical generative model

Humans have long wondered about the function of mental imagery and its relationship to vision. Although visual representations are utilized during imagery, the computations they subserve are unclear. Building on a theory that treats vision as inference about the causes of sensory stimulation in an internal generative model, we propose that mental imagery is inference about the sensory consequences of predicted or remembered causes. The relation between these complementary inferences yields a relation between the brain activity patterns associated with imagery and vision. We show that this relation has the formal structure of an echo that makes encoding of imagined stimuli in low-level visual areas resemble the encoding of seen stimuli in higher areas. To test for evidence of this echo effect we developed imagery encoding models--a new tool for revealing how imagined stimuli are encoded in brain activity. We estimated imagery encoding models from brain activity measured with fMRI while human subjects imagined complex visual stimuli, and then compared these to visual encoding models estimated from a matched viewing experiment. Consistent with an echo effect, imagery encoding models in low-level visual areas exhibited decreased spatial frequency preference and larger, more foveal receptive fields, thus resembling visual encoding models in high-level visual areas where imagery and vision appeared to be almost interchangeable. Our findings support an interpretation of mental imagery as a predictive inference that is conditioned on activity in high-level visual cortex, and is related to vision through shared dependence on an internal model of the visual world.

neuroscience↗

bioRxiv2018

Generative Adversarial Networks Conditioned on Brain Activity Reconstruct Seen Images

We consider the inference problem of reconstructing a visual stimulus from brain activity measurements (e.g. fMRI) that encode this stimulus. Recovering a complete image is complicated by the fact that neural representations are noisy, high-dimensional, and contain incomplete information about image details. Thus, reconstructions of complex images from brain activity require a strong prior. Here we propose to train generative adversarial networks (GANs) to learn a generative model of images that is conditioned on measurements of brain activity. We consider two challenges of this approach: First, given that GANs require far more data to train than is typically collected in an fMRI experiment, how do we obtain enough samples to train a GAN that is conditioned on brain activity? Secondly, how do we ensure that our generated samples are robust against noise present in fMRI data? Our strategy to surmount both of these problems centers around the creation of surrogate brain activity samples that are generated by an encoding model. We find that the generative model thus trained generalizes to real fRMI data measured during perception of images and is able to reconstruct the basic outline of the stimuli.

neuroscience↗

bioRxiv2017

The feature-weighted receptive field: an interpretable encoding model for complex feature spaces.

We introduce the feature-weighted receptive field (fwRF), an encoding model designed to balance expressiveness, interpretability and scalability. The fwRF is organized around the notion of a feature map--a transformation of visual stimuli into visual features that preserves the topology of visual space (but not necessarily the native resolution of the stimulus). The key assumption of the fwRF model is that activity in each voxel encodes variation in a spatially localized region across multiple feature maps. This region is fixed for all feature maps; however, the contribution of each feature map to voxel activity is weighted. Thus, the model has two separable sets of parameters: \"where\" parameters that characterize the location and extent of pooling over visual features, and \"what\" parameters that characterize tuning to visual features. The \"where\" parameters are analogous to classical receptive fields, while \"what\" parameters are analogous to classical tuning functions. By treating these as separable parameters, the fwRF model complexity is independent of the resolution of the underlying feature maps. This makes it possible to estimate models with thousands of high-resolution feature maps from relatively small amounts of data. Once a fwRF model has been estimated from data, spatial pooling and feature tuning can be read-off directly with no (or very little) additional post-processing or in-silico experimentation.\n\nWe describe an optimization algorithm for estimating fwRF models from data acquired during standard visual neuroimaging experiments. We then demonstrate the models application to two distinct sets of features: Gabor wavelets and features supplied by a deep convolutional neural network. We show that when Gabor feature maps are used, the fwRF model recovers receptive fields and spatial frequency tuning functions consistent with known organizational principles of the visual cortex. We also show that a fwRF model can be used to regress entire deep convolutional networks against brain activity. The ability to use whole networks in a single encoding model yields state-of-the-art prediction accuracy. Our results suggest a wide variety of uses for the feature-weighted receptive field model, from retinotopic mapping with natural scenes, to regressing the activities of whole deep neural networks onto measured brain activity.

neuroscience↗