Biology subjects

Tarr, M. J.

Publications and source records attributed to Tarr, M. J..

12 recordsLinked to original sources

bioRxiv2026

Diffusion-based stimulus optimization reveals functional organization across higher visual cortex

Characterizing the fine-grained functional organization of human higher visual cortex remains a central challenge, as traditional neuroimaging experiments constrain the diversity of stimuli that can be sampled. In prior work we addressed this challenge by developing a novel data-driven tool, termed "BrainDiVE" (Luo et al. 2023), which synthesizes naturalistic images predicted to strongly activate specific brain regions. BrainDiVE leverages pretrained image diffusion models guided by gradients from an image-computable fMRI encoding model. Here, we experimentally validated BrainDiVE by generating images predicted to maximally activate different functional regions of interest and then presenting them to new participants (n=12) in an fMRI study. The model-generated images elicited robust, spatially specific responses in the targeted brain regions, producing significantly greater category selectivity than natural images, validating the methods ability to capture generalizable neural tuning properties in human ventral visual cortex. We further showed that region-targeted images exaggerate specific sets of low-level and mid-level image statistics, suggesting that category-selective regions are tuned to continuous directions in feature space. Moreover, we demonstrated fine-grained experimental control by differentially activating two face-selective regions, the occipital face area (OFA) and fusiform face area (FFA), providing additional evidence that these regions encode distinct aspects of faces. Finally, we identify a posterior-to-anterior functional gradient within the occipital place area (OPA), suggesting topographic organization based on scene properties such as distance and indoor-outdoor location. These findings enhance our understanding of the representational structure of category-selective regions and introduce a new paradigm for probing neural selectivity in human visual cortex.

neuroscience↗

bioRxiv2023

Brain Dissection: fMRI-trained Networks Reveal Spatial Selectivity in the Processing of Natural Images

The alignment between deep neural network (DNN) features and cortical responses currently provides the most accurate quantitative explanation for higher visual areas [1, 2, 3, 4]. At the same time, these model features have been critiqued as uninterpretable explanations, trading one black box (the human brain) for another (a neural network). In this paper, we train networks to directly predict, from scratch, brain responses to images from a large-scale dataset of natural scenes [5]. We then use "network dissection" [6], an explainable AI technique used for enhancing neural network interpretability by identifying and localizing the most significant features in images for individual units of a trained network, and which has been used to study category selectivity in the human brain [7]. We adapt this approach to create a hypothesis-neutral model that is then used to explore the tuning properties of specific visual regions beyond category selectivity, which we call "brain dissection". We use brain dissection to examine a range of ecologically important, intermediate properties, including depth, surface normals, curvature, and object relations across sub-regions of the parietal, lateral, and ventral visual streams, and scene-selective regions. Our findings reveal distinct preferences in brain regions for interpreting visual scenes, with ventro-lateral areas favoring closer and curvier features, medial and parietal areas opting for more varied and flatter 3D elements, and the parietal region uniquely preferring spatial relations. Scene-selective regions exhibit varied preferences, as the retrosplenial complex prefers distant and outdoor features, while the occipital and parahippocampal place areas favor proximity, verticality, and in the case of the OPA, indoor elements. Such findings show the potential of using explainable AI to uncover spatial feature selectivity across the visual cortex, contributing to a deeper, more fine-grained understanding of the functional characteristics of human visual cortex when viewing natural scenes.

neuroscience↗

bioRxiv2022

Incorporating natural language into vision models improves prediction and understanding of higher visual cortex

Advances in neural networks have been catalyzed by joint training on images and natural language, increased dataset sizes, and data diversity. We explored whether the same factors support similar improvements in predicting visual responses in the human brain. We used models pre-trained with Contrastive Language-Image Pre-training (CLIP) - which learns image embeddings that best match text embeddings of image captions from diverse, large-scale datasets - to study visual representations. We built voxelwise encoding models based on CLIP image features to predict brain responses to real-world images. ResNet50 with CLIP explained up to R2 = 79% of variance in individual voxel responses in held-out test data, a significant increase from models trained only with image/label pairs (ImageNet trained ResNet) or text (BERT). Comparisons across different model backbones ruled out network architecture as a factor in performance improvements. Comparisons across models that controlled for dataset size and data diversity demonstrated that language feedback along with data diversity in larger datasets are important factors in explaining neural responses in high-level visual brain regions. Visualizations of model embeddings and Principal Component Analysis (PCA) revealed that our models capture both global and fine-grained semantic dimensions represented within human visual cortex.

neuroscience↗

bioRxiv2022

A texture statistics encoding model reveals hierarchical feature selectivity across human visual cortex

Mid-level visual features, such as contour and texture, provide a computational link between low- and high-level visual representations. While the detailed nature of mid-level representations in the brain is not yet fully understood, past work has suggested that a texture statistics model (P-S model; Portilla and Simoncelli, 2000) is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex regions to natural scene images. To examine this, we constructed single voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (male and female) from the Natural Scenes Dataset (Allen et al., 2021). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas as well as higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex voxels suggests that the representation of texture statistics features is widespread throughout the brain, potentially playing a role in higher-order processes like object recognition. Furthermore, we use variance partitioning analyses to identify which features are most uniquely predictive of brain responses, and show that the contributions of higher-order texture features increases from early areas to higher areas on the ventral and lateral surface of the brain. These results provide a key step forward in characterizing how mid-level feature representations emerge hierarchically across the visual system. Significance StatementIntermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.

neuroscience↗

bioRxiv2022

Low-level tuning biases in higher visual cortex reflect the semantic informativeness of visual features

Representations of visual and semantic information can overlap in human visual cortex, with the same neural populations exhibiting sensitivity to low-level features (orientation, spatial frequency, retinotopic position), and high-level semantic categories (faces, scenes). It has been hypothesized that this relationship between low-level visual and high-level category neural selectivity reflects natural scene statistics, such that neurons in a given category-selective region are tuned for low-level features or spatial positions that are diagnostic of the regions preferred category. To address the generality of this "natural scene statistics" hypothesis, as well as how well it can account for responses to complex naturalistic images across visual cortex, we performed two complementary analyses. First, across a large set of rich natural scene images, we demonstrated reliable associations between low-level (Gabor) features and high-level semantic dimensions (indoor-outdoor, animacy, real-world size), with these relationships varying spatially across the visual field. Second, we used a large-scale fMRI dataset (the Natural Scenes Dataset) and a voxelwise forward encoding model to estimate the feature and spatial selectivity of neural populations throughout visual cortex. We found that voxels in category-selective visual regions exhibit systematic biases in their feature and spatial selectivity which are consistent with their hypothesized roles in category processing. We further showed that these low-level tuning biases are largely independent of viewed image category. Together, our results are consistent with a framework in which low-level feature selectivity contributes to the computation of high-level semantic category information in the brain.

neuroscience↗

bioRxiv2022

Why is human vision so poor in early development? The impact of initial sensitivity to low spatial frequencies on visual category learning

Humans are born with very low contrast sensitivity, meaning that developing infants experience the world "in a blur". Is this solely a byproduct of maturational processes or is there some functional advantage for beginning life with poor vision? We explore whether reduced visual acuity as a consequence of low contrast sensitivity facilitates the acquisition of basic-level visual categories and, if so, whether this advantage also enhances subordinate-level category learning as visual acuity improves. Using convolutional neural networks (CNNs) and the ecoset dataset to simulate basic-level category learning, we manipulated model training curricula along three dimensions: presence of blurred inputs early in training, rate of blur removal over time, and grayscale versus color inputs. We found that a training regimen where blur starts high and is gradually reduced over time - as in human development - improves basic-level categorization performance relative to a regimen in which non-blurred inputs are used throughout. However, this pattern was observed only when grayscale images were used (analogous to the low sensitivity to color infants experience during early development). Importantly, the observed improvements in basic-level performance generalized to subordinate-level categorization as well: when models were fine-tuned on a dataset including subordinate-level categories (ImageNet), we found that models initially trained with blurred inputs showed a greater performance benefit than models trained solely on non-blurred inputs. Consistent with several other recent studies, we conclude that poor visual acuity in human newborns confers multiple advantages, including, as demonstrated here, more rapid and accurate acquisition of visual object categories at multiple hierarchical levels. Author SummaryWhy do humans start life with extremely poor vision? The common evolutionary story is that head size is small to accommodate the development of human-level intelligence. However, there is growing evidence that beginning life in a premature state confers short-term advantages. The "starting small" principle states that learning can be facilitated by restricted or impoverished inputs that reduce the learning complexity. We suggest that blurred vision during early development biases learning toward shape features, which organize objects naturally into "basic-level" categories that are the foundation of human cognition (e.g., birds, cars, etc.). Such categories are defined by their visual similarity in global object shape. Since blurring restricts visual inputs to emphasize shape information, it should enhance the learning of basic-level categories. We trained artificial neural-network models on basic-level categorization using either blurred or non-blurred color or grayscale images and found that models trained with blurred images outperformed models trained with non-blurred images, but only for grayscale. These same models performed better in subsequent "subordinate-level" tasks that required discriminating between objects within a basic-level category. Our simulations provide evidence that initially poor vision in infants has an important functional role in organizing knowledge about complex environments.

neuroscience↗

bioRxiv2022

Food for thought: selectivity for food in human ventral visual cortex

Visual cortex contains regions of selectivity for domains of ecological importance. Food is an evolutionarily critical category whose visual heterogeneity may make the identification of selectivity more challenging. We investigate neural responsiveness to food using natural images combined with large-scale human fMRI. Leveraging the improved sensitivity of modern designs and statistical analyses, we identify two food-selective regions in the ventral visual cortex. Our results are robust across 8 subjects from the Natural Scenes Dataset (NSD), multiple independent image sets and multiple analysis methods. We then test our findings of food selectivity in an fMRI "localizer" using grayscale food images. These independent results confirm the existence of food selectivity in ventral visual cortex and help illuminate why earlier studies may have failed to do so. Our identification of food-selective regions stands alongside prior findings of functional selectivity and adds to our understanding of the organization of knowledge within the human visual system.

neuroscience↗

bioRxiv2022

GLMsingle: a toolbox for improving single-trial fMRI response estimates

Advances in modern artificial intelligence (AI) have inspired a paradigm shift in human neuroscience, yielding large-scale functional magnetic resonance imaging (fMRI) datasets that provide high-resolution brain responses to tens of thousands of naturalistic visual stimuli. Because such experiments necessarily involve brief stimulus durations and few repetitions of each stimulus, achieving sufficient signal-to-noise ratio can be a major challenge. We address this challenge by introducing GLMsingle, a scalable, user-friendly toolbox available in MATLAB and Python that enables accurate estimation of single-trial fMRI responses (glmsingle.org). Requiring only fMRI time-series data and a design matrix as inputs, GLMsingle integrates three techniques for improving the accuracy of trial-wise general linear model (GLM) beta estimates. First, for each voxel, a custom hemodynamic response function (HRF) is identified from a library of candidate functions. Second, cross-validation is used to derive a set of noise regressors from voxels unrelated to the experimental paradigm. Third, to improve the stability of beta estimates for closely spaced trials, betas are regularized on a voxel-wise basis using ridge regression. Applying GLMsingle to the Natural Scenes Dataset and BOLD5000, we find that GLMsingle substantially improves the reliability of beta estimates across visually-responsive cortex in all subjects. Furthermore, these improvements translate into tangible benefits for higher-level analyses relevant to systems and cognitive neuroscience. Specifically, we demonstrate that GLMsingle: (i) improves the decorrelation of response estimates between trials that are nearby in time; (ii) enhances representational similarity between subjects both within and across datasets; and (iii) boosts one-versus-many decoding of visual stimuli. GLMsingle is a publicly available tool that can significantly improve the quality of past, present, and future neuroimaging datasets that sample brain activity across many experimental conditions.

neuroscience↗

bioRxiv2020

Functional Context Affects Scene Processing

Rapid visual perception is often viewed as a bottom-up process. Category-preferred neural regions are often characterized as automatic, default processing mechanisms for visual inputs of their categorical preference. To explore the sensitivity of such regions to top-down information, we examined three scene-preferring brain regions, the occipital place area (OPA), the parahippocampal place area (PPA), and the retrosplenial complex (RSC), and tested whether the processing of outdoor scenes is influenced by the functional contexts in which they are seen. Context was manipulated by presenting real-world landscape images as if being viewed through a window or within a picture frame; manipulations that do not affect scene content but do affect ones functional knowledge regarding the scene. This manipulation influences neural scene processing (as measured by fMRI): the OPA and PPA exhibited greater neural activity when participants viewed images as if through a window as compared to within a picture frame, while the RSC did not show this difference. In a separate behavioral experiment, functional context affected scene memory in predictable directions (boundary extension). Our interpretation is that the window context denotes three-dimensionality, therefore rendering the perceptual experience of viewing landscapes as more realistic. Conversely, the frame context denotes a two-dimensional image. As such, more spatially-biased scene representations in the OPA and the PPA are influenced by differences in top-down, perceptual expectations generated from context. In contrast, more semantically-biased scene representations in the RSC are likely to be less affected by top-down signals that carry information about the physical layout of a scene.

neuroscience↗

bioRxiv2019

Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity

Convolutional neural networks (CNNs) trained for object recognition have been widely used to account for visually-driven neural responses in both the human and primate brains. However, because of the generality and complexity of the task of object classification, it is often difficult to make precise inferences about neural information processing using CNN representations from object classification despite the fact that these representations are effective for predicting brain activity. To better understand underlying the nature of the visual features encoded in different brain regions of the human brain, we predicted brain responses to images using fine-grained representations drawn from 19 specific computer vision tasks. Individual encoding models for each task were constructed and then applied to BOLD5000--a large-scale dataset comprised of fMRI scans collected while observers viewed over 5000 naturalistic scene and object images. Because different encoding models predict activity in different brain regions, we were able to associate specific vision tasks with each region. For example, within scene-selective brain regions, features from 3D tasks such as 3D keypoints and 3D edges explain greater variance as compared to 2D tasks--a pattern that replicates across the whole brain. Using results across all 19 task representations, we constructed a \"task graph\" based on the spatial layout of well-predicted brain areas from each task. We then compared the brain-derived task structure with the task structure derived from transfer learning accuracy in order to assess the degree of shared information between the two task spaces. These computationally-driven results--arising out of state-of-the-art computer vision methods--begin to reveal the task-specific architecture of the human visual system.

neuroscience↗

bioRxiv2018

Exploring spatio-temporal neural dynamics of the human visual cortex

The human visual cortex is organized in a hierarchical manner. Although a significant body of evidence has been accumulated in support of this hypothesis, specific details regarding the spatial and temporal information flow remain open. Here we present detailed spatio-temporal correlation profiles of neural activity with low-level and high-level features derived from a \"deep\" (8-layer) neural network pre-trained for object recognition. These correlation profiles indicate an early-to-late shift from low-level features to high-level features and from low-level regions to higher-level regions along the visual hierarchy, consistent with feedforward information flow. To refine our understanding of information flow, we computed three sets of features from the low-and high-level features provided by the neural network: object-category-relevant low-level features (the common components between low-level and high-level features), low-level features roughly orthogonal to high-level features (the residual Layer 1 features), and unique high-level features that were roughly orthogonal to low-level features (the residual Layer 7 features). Contrasting the correlation effects of the common components and the residual Layer 1 features, we observed that the early visual cortex exhibits a similar amount of correlation with the two feature sets early in time (60 to 120 ms), but in a later time window, the early visual cortex exhibits a higher and longer correlation effect with the common components/low-level task-relevant features as compared to the low-level residual features--an effect unlikely to arise from purely feedforward information flow. Overall, our results indicate that non-feedforward processes, for example, top-down influences from mental representations of categories, may facilitate differentiation between these two types of low-level features within the early visual cortex.

neuroscience↗

bioRxiv2017

Very high density EEG elucidates spatiotemporal aspects of early visual processing

Standard human EEG systems based on spatial Nyquist estimates suggest that 20-30 mm electrode spacing suffices to capture neural signals on the scalp, but recent studies posit that increasing sensor density can provide higher resolution neural information. Here, we compared \"super-Nyquist\" density EEG (\"SND\") with Nyquist density (\"ND\") arrays for assessing the spatiotemporal aspects of early visual processing. EEG was measured from 128 electrodes arranged over occipitotemporal brain regions (14 mm spacing) while participants viewed flickering checkerboard stimuli. Analyses compared SND with ND-equivalent subsets of the same electrodes. Frequency-tagged stimuli were classified more accurately with SND than ND arrays in both the time and the frequency domains. Representational similarity analysis revealed that a computational model of V1 correlated more highly with the SND than the ND array. Overall, SND EEG captured more neural information from visual cortex, arguing for increased development of this approach in basic and translational neuroscience.\n\nAbbreviations

neuroscience↗