Dipani, A. | Search bioRxiv

bioRxiv2026

Zero-shot evaluations expose structured generalization limits in predictive models of the brain

Artificial neural networks (ANNs) can predict neural responses to natural images with remarkable accuracy, making them promising models of human vision. However, prediction scores alone reveal little about what computations these models have captured or how well they generalize beyond the images on which they are trained. Here, we introduce a principled zero-shot model evaluation framework in which ANN-based encoding models are frozen before being tested on not just held out images but entirely new datasets. Our strongest tests repurpose decades of cognitive neuroscience experiments as diagnostic benchmarks for identifying which computations current models capture and where they fail. To implement this framework we built encoding models of human category-selective regions (FFA, PPA, and EBA) using several pretrained ANNs and evaluated these fixed models across 7 independent fMRI datasets and 31 cognitive experiments from 10 published studies. This framework revealed three broad patterns. First, zero-shot evaluations exposed systematic limits to model predictivity that were largely hidden by standard within-dataset cross-validation. These limits were structured varying across brain regions and model classes and depending strongly on the similarity between the images used to build and test the models. Second, current models reproduced some, but not all, classical findings from cognitive neuroscience, providing a diagnosis of the computations that current models have yet to capture. Finally, models that performed well on prediction tests also tended to succeed at reproducing cognitive neuroscience findings, suggesting that prediction and explanation are closely linked. Unlike prediction scores however, cognitive tests help diagnose where and why models fail. Together, the zero-shot evaluation framework provides a scalable and unified approach for evaluating brain models across prediction and cognitive neuroscience tests, which can help us better understand what current models capture and what still remains to be explained.

neuroscience↗

bioRxiv2026

Category selectivity observed in the human brain is distinct from category selectivity observed in artificial neural networks

Category selectivity for images of faces, scenes, and bodies is among the most striking and reproducible findings in vision neuroscience. Artificial neural networks (ANNs) trained on visual tasks also develop category-selective units, which has led to the suggestion that ANNs may capture important aspects of how the brain processes visual categories. But the mere presence of category-selective units in ANNs does not mean that those units are selective in the same way as the brain. Here, we distinguish between the presence of category-selective units in ANNs from the form of selectivity they express, and show that the selectivity that emerges in ANN units differs in meaningful and systematic ways from that observed in the human brain with fMRI. To this end, we first identified category-selective units in a wide range of ANN models using standard fMRI localizers, and found that selective units emerged reliably in trained, but not in untrained, ANNs. We then identified category-selective regions in the human brain using the same localizer and found that their response tuning to a broad range of images was strikingly consistent across individuals. Thus, category-selective regions exhibit a stable representational signature shared across subjects. Category-selective ANN units did not match this structure. Their responses diverged in both univariate tuning and multivariate representational geometry, fell well below the human-human ceiling, varied substantially across models, and depended strongly on the localizer used to identify them. We also found that the category-selective ANN units were neither necessary nor sufficient for predicting neural responses using an encoding model. Further stimulus-level analyses revealed clear and interpretable mismatches between ANN selectivity and human fMRI responses, which can be used to test and compare better ANN models in the future. Taken together, these results show that the full range of response tuning in category-selective regions provides a highly demanding and discriminative test of brain-model alignment than previously appreciated. Although current ANNs contain category-selective units, the selectivity they express is more fragile and does not capture the stable and shared form of selectivity observed in the human brain.

neuroscience↗