Biology subjects

Conwell, C.

Publications and source records attributed to Conwell, C..

4 recordsLinked to original sources

bioRxiv2025

Monkey See, Model Knew: Large Language Models Accurately Predict Visual Brain Responses in Humans and Non-Human Primates

AO_SCPLOWBSTRACTC_SCPLOWRecent progress in multimodal AI and language-aligned visual representation learning has rekindled debates about the role of language in shaping the human visual system. In particular, the emergent ability of language-aligned vision models (e.g. CLIP) - and even pure language models (e.g. BERT) - to predict image-evoked brain activity has led some to suggest that human visual cortex itself may be language-aligned in comparable ways. But what would we make of this claim if the same procedures could model visual activity in a species without language? Here, we conducted controlled comparisons of pure-vision, pure-language, and multimodal vision-language models in their prediction of human (N=4) and rhesus macaque (N=6, 5:IT, 1:V1) ventral visual activity to the same set of 1000 captioned natural images (the NSD1000). The results revealed markedly similar patterns in model predictivity of early and late ventral visual cortex across both species. This suggests that language model predictivity of the human visual system is not necessarily due to the evolution or learning of language perse, but rather to the statistical structure of the visual world that is reflected in natural language.

neuroscience↗

bioRxiv2024

Universality of representation in biological and artificial neural networks

Many artificial neural networks (ANNs) trained with ecologically plausible objectives on naturalistic data align with behavior and neural representations in biological systems. Here, we show that this alignment is a consequence of convergence onto the same representations by high-performing ANNs and by brains. We developed a method to identify stimuli that systematically vary the degree of inter-model representation agreement. Across language and vision, we then showed that stimuli from high-and low-agreement sets predictably modulated model-to-brain alignment. We also examined which stimulus features distinguish high-from low-agreement sentences and images. Our results establish representation universality as a core component in the model-to-brain alignment and provide a new approach for using ANNs to uncover the structure of biological representations and computations.

neuroscience↗

bioRxiv2022

Large-Scale Benchmarking of Diverse Artificial Vision Models in Prediction of 7T Human Neuroimaging Data

The rapid development and open-source release of highly performant computer vision models offers new potential for examining how different inductive biases impact representation learning and emergent alignment with the high-level human ventral visual system. Here, we assess a diverse set of 224 models, curated to enable controlled comparison of different model properties, testing their brain predictivity using large-scale functional magnetic resonance imaging data. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and markedly different task objectives (e.g. purely visual contrastive learning versus vision-language alignment) achieve near equivalent degrees of brain predictivity, when other factors are held constant. Instead, variation across model visual training diets yields the largest, most consistent effect on emergent brain predictivity. Overarching model properties commonly suspected to increase brain predictivity (e.g. greater effective dimensionality; learnable parameter count) were not robust indicators across this more extensive survey. We highlight that standard model-to-brain linear re-weighting methods may be too flexible, as most performant models have very similar brain-predictivity scores, despite significant variation in their underlying representations. Broadly, our findings point to the importance of visual diet, challenge common assumptions about the methods used to link models to brains, and more concretely outline future directions for leveraging the full diversity of existing open-source models as tools to probe the common computational principles underlying biological and artificial visual systems.

neuroscience↗

bioRxiv2021

Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex

How well do deep neural networks fare as models of mouse visual cortex? A majority of research to date suggests results far more mixed than those produced in the modeling of primate visual cortex. Here, we perform a large-scale benchmarking of dozens of deep neural network models in mouse visual cortex with both representational similarity analysis and neural regression. Using the Allen Brain Observatorys 2-photon calcium-imaging dataset of activity in over 6,000 reliable rodent visual cortical neurons recorded in response to natural scenes, we replicate previous findings and resolve previous discrepancies, ultimately demonstrating that modern neural networks can in fact be used to explain activity in the mouse visual cortex to a more reasonable degree than previously suggested. Using our benchmark as an atlas, we offer preliminary answers to overarching questions about levels of analysis (e.g. do models that better predict the representations of individual neurons also predict representational similarity across neural populations?); questions about the properties of models that best predict the visual system overall (e.g. is convolution or category-supervision necessary to better predict neural activity?); and questions about the mapping between biological and artificial representations (e.g. does the information processing hierarchy in deep nets match the anatomical hierarchy of mouse visual cortex?). Along the way, we catalogue a number of models (including vision transformers, MLP-Mixers, normalization free networks, Taskonomy encoders and self-supervised models) outside the traditional circuit of convolutional object recognition. Taken together, our results provide a reference point for future ventures in the deep neural network modeling of mouse visual cortex, hinting at novel combinations of mapping method, architecture, and task to more fully characterize the computational motifs of visual representation in a species so central to neuroscience, but with a perceptual physiology and ecology markedly different from the ones we study in primates.

neuroscience↗