Biology subjects

St-Yves, G.

Publications and source records attributed to St-Yves, G..

8 recordsLinked to original sources

bioRxiv2025

A transformation from vision to imagery in the human brain

Extensive work has shown that the visual cortex is reactivated during mental imagery, and that models trained on visual data can predict imagery activity and decode imagined stimuli. These findings may explain why imagery can feel and function like vision, but give little insight into how the brain activity patterns that encode seen and mental images differ. While one popular theory ("weak vision") suggests that imagery differs from vision only in strength, recent work points to more complex differences. To clarify the relationship between visual and imagery activity in the human brain, we introduce the concept of an imagery transformation--a mapping from visual to imagery activity patterns evoked by the same stimulus. Importantly, this approach can describe a variety of possible scenarios, from simple rescaling to the selective removal or reorientation of activity dimensions. Using two 7T fMRI datasets, we estimated imagery transformations across different visual areas and found they accurately predict imagery activity. We show that imagery transformations are indeed more complex than simple weakening: in early visual cortex, they halve the number of active dimensions and reorient them, such that reconstructions of visual activity in terms of imagery dimensions explain only 25-50% of the variance. Nonetheless, these reoriented dimensions still coarsely encode the features encoded by the principal dimensions of visual activity. These findings help to explain the "same but different" relationship between mental imagery and vision: imagery activity patterns are transformations of visual activity patterns that approximate them, but encode fewer features and occupy a distinct subspace within the overall space of brain activations.

neuroscience↗

bioRxiv2024

Variation in the geometry of concept manifolds acrosshuman visual cortex

Brain activity patterns in high-level visual cortex support accurate linear classification of visual concepts (e.g., objects or scenes). It has long been appreciated that the accuracy of linear classification in any brain area depends on the geometry of its concept manifolds--sets of brain activity patterns that encode images of a concept. However, it is unclear how the geometry of concept manifolds differs between regions of visual cortex that support accurate classification and those that dont, or how it differs between visual cortex and deep neural networks (DNNs). We estimated geometric properties of concept manifolds that, per a recent theory, directly determine the accuracy of simple "few-shot" linear classifiers. Using a large fMRI dataset, we show that variation in classification accuracy across human visual cortex is driven by a variation in a single geometric property: the distance between manifold centers ("geometric Signal"). In contrast, variation in classification accuracy across most DNN layers is driven by an increase in the effective number of manifold dimensions ("Dimensionality"). Despite this difference in the geometric properties that affect few-shot classification performance in the brain and DNNs, we find that Signal and Dimensionality are strongly, negatively correlated: when Signal increases across brain regions or DNN layers, Dimensionality decreases, and vice versa. We conclude that visual cortex and DNNs deploy different geometric strategies for accurate linear classification of concepts, even though both are subject to the same constraint.

neuroscience↗

bioRxiv2022

Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex

Deep neural networks (DNNs) trained to perform visual tasks learn representations that align with the hierarchy of visual areas in the primate brain. This finding has been taken to imply that the primate visual system forms representations by passing them through a hierarchical sequence of brain areas, just as DNNs form representations by passing them through a hierarchical sequence of layers. To test the validity of this assumption, we optimized DNNs not to perform visual tasks but to directly predict brain activity in human visual areas V1-V4. Using a massive sampling of human brain activity, we constructed brain-optimized networks that predict brain activity even more accurately than task-optimized networks. We show that brain-optimized networks can learn representations that diverge from those formed in a strict hierarchy. Brain-optimized networks do not need to align representations in V1-V4 with layer depth; moreover, they are able to accurately model anterior brain areas (e.g., V4) without computing intermediary representations associated with posterior brain areas (e.g., V1). Our results challenge the view that human visual areas V1-V4 act--like the early layers of a DNN--as a serial pre-processing sequence for higher areas, and suggest they may subserve their own independent functions.

neuroscience↗

bioRxiv2021

A massive 7T fMRI dataset to bridge cognitive and computational neuroscience

Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. We present the Natural Scenes Dataset (NSD), in which high-resolution fMRI responses to tens of thousands of richly annotated natural scenes are measured while participants perform a continuous recognition task. To optimize data quality, we develop and apply novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we use NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality, and breadth, NSD opens new avenues of inquiry in cognitive and computational neuroscience.

neuroscience↗

bioRxiv2019

Voxel-to-voxel predictive models reveal unexpected structure in unexplained variance

Encoding models based on deep convolutional neural networks (DCNN) more accurately predict BOLD responses to natural scenes in the visual system than any other currently available model. However, DCNN-based encoding models fail to predict a significant amount of variance in the activity of most voxels in all visual areas. This failure could reflect limitations in the data (e.g., a noise ceiling), or could reflect limitations of the DCNN as a model of computation in the brain. Understanding the source and structure of the unexplained variance could therefore provide helpful clues for improving models of brain computation. Here, we characterize the structure of the variance that DCNN-based encoding models cannot explain. Using a publicly available dataset of BOLD responses to natural scenes, we determined if the source of unexplained variance was shared across voxels, individual brains, retinotopic locations, and hierarchically distant visual brain areas. We answered these questions using voxel-to-voxel (vox2vox) models that predict activity in a target voxel given activity in a population of source voxels. We found that simple linear vox2vox models increased within-subject prediction accuracy over DCNN-based models for any pair of source/target visual areas, clearly demonstrating that the source of unexplained variance is widely shared within and across visual brain areas. However, vox2vox models were not more accurate than DCNN-based models when source and target voxels came from separate brains, demonstrating that the source of unexplained variance was not shared across brains. Furthermore, the weights of these vox2vox models permitted explicit readout of the receptive field location of target voxels, demonstrating that the source of unexplained variance induces correlations primarily between the activities of voxels with overlapping receptive fields. Finally, we found that vox2vox model prediction accuracy was heavily dependent upon the signed hierarchical distance between the source and target voxels: for feed-forward models (source area lower in the visual hierarchy than target area) prediction accuracy decreased with hierarchical distance between source and target. It did not decrease for feedback models. In contrast, the same analysis applied across layers of a DCNN did not reveal this feed-forward/feedback asymmetry. Given these results, we argue that the structured variance unexplained by DCNN-based encoding models is unlikely to be entirely caused by spatially correlated noise or eye movements; rather, our results point to a need for brain models that include endogenous dynamics and a pattern of connectivity that is not strictly feed-forward.

neuroscience↗

bioRxiv2018

Human brain activity during mental imagery exhibits signatures of inference in a hierarchical generative model

Humans have long wondered about the function of mental imagery and its relationship to vision. Although visual representations are utilized during imagery, the computations they subserve are unclear. Building on a theory that treats vision as inference about the causes of sensory stimulation in an internal generative model, we propose that mental imagery is inference about the sensory consequences of predicted or remembered causes. The relation between these complementary inferences yields a relation between the brain activity patterns associated with imagery and vision. We show that this relation has the formal structure of an echo that makes encoding of imagined stimuli in low-level visual areas resemble the encoding of seen stimuli in higher areas. To test for evidence of this echo effect we developed imagery encoding models--a new tool for revealing how imagined stimuli are encoded in brain activity. We estimated imagery encoding models from brain activity measured with fMRI while human subjects imagined complex visual stimuli, and then compared these to visual encoding models estimated from a matched viewing experiment. Consistent with an echo effect, imagery encoding models in low-level visual areas exhibited decreased spatial frequency preference and larger, more foveal receptive fields, thus resembling visual encoding models in high-level visual areas where imagery and vision appeared to be almost interchangeable. Our findings support an interpretation of mental imagery as a predictive inference that is conditioned on activity in high-level visual cortex, and is related to vision through shared dependence on an internal model of the visual world.

neuroscience↗

bioRxiv2018

Generative Adversarial Networks Conditioned on Brain Activity Reconstruct Seen Images

We consider the inference problem of reconstructing a visual stimulus from brain activity measurements (e.g. fMRI) that encode this stimulus. Recovering a complete image is complicated by the fact that neural representations are noisy, high-dimensional, and contain incomplete information about image details. Thus, reconstructions of complex images from brain activity require a strong prior. Here we propose to train generative adversarial networks (GANs) to learn a generative model of images that is conditioned on measurements of brain activity. We consider two challenges of this approach: First, given that GANs require far more data to train than is typically collected in an fMRI experiment, how do we obtain enough samples to train a GAN that is conditioned on brain activity? Secondly, how do we ensure that our generated samples are robust against noise present in fMRI data? Our strategy to surmount both of these problems centers around the creation of surrogate brain activity samples that are generated by an encoding model. We find that the generative model thus trained generalizes to real fRMI data measured during perception of images and is able to reconstruct the basic outline of the stimuli.

neuroscience↗

bioRxiv2017

The feature-weighted receptive field: an interpretable encoding model for complex feature spaces.

We introduce the feature-weighted receptive field (fwRF), an encoding model designed to balance expressiveness, interpretability and scalability. The fwRF is organized around the notion of a feature map--a transformation of visual stimuli into visual features that preserves the topology of visual space (but not necessarily the native resolution of the stimulus). The key assumption of the fwRF model is that activity in each voxel encodes variation in a spatially localized region across multiple feature maps. This region is fixed for all feature maps; however, the contribution of each feature map to voxel activity is weighted. Thus, the model has two separable sets of parameters: \"where\" parameters that characterize the location and extent of pooling over visual features, and \"what\" parameters that characterize tuning to visual features. The \"where\" parameters are analogous to classical receptive fields, while \"what\" parameters are analogous to classical tuning functions. By treating these as separable parameters, the fwRF model complexity is independent of the resolution of the underlying feature maps. This makes it possible to estimate models with thousands of high-resolution feature maps from relatively small amounts of data. Once a fwRF model has been estimated from data, spatial pooling and feature tuning can be read-off directly with no (or very little) additional post-processing or in-silico experimentation.\n\nWe describe an optimization algorithm for estimating fwRF models from data acquired during standard visual neuroimaging experiments. We then demonstrate the models application to two distinct sets of features: Gabor wavelets and features supplied by a deep convolutional neural network. We show that when Gabor feature maps are used, the fwRF model recovers receptive fields and spatial frequency tuning functions consistent with known organizational principles of the visual cortex. We also show that a fwRF model can be used to regress entire deep convolutional networks against brain activity. The ability to use whole networks in a single encoding model yields state-of-the-art prediction accuracy. Our results suggest a wide variety of uses for the feature-weighted receptive field model, from retinotopic mapping with natural scenes, to regressing the activities of whole deep neural networks onto measured brain activity.

neuroscience↗