Al-Tahan, H. | Search bioRxiv

bioRxiv2026

Zero-shot evaluations expose structured generalization limits in predictive models of the brain

Artificial neural networks (ANNs) can predict neural responses to natural images with remarkable accuracy, making them promising models of human vision. However, prediction scores alone reveal little about what computations these models have captured or how well they generalize beyond the images on which they are trained. Here, we introduce a principled zero-shot model evaluation framework in which ANN-based encoding models are frozen before being tested on not just held out images but entirely new datasets. Our strongest tests repurpose decades of cognitive neuroscience experiments as diagnostic benchmarks for identifying which computations current models capture and where they fail. To implement this framework we built encoding models of human category-selective regions (FFA, PPA, and EBA) using several pretrained ANNs and evaluated these fixed models across 7 independent fMRI datasets and 31 cognitive experiments from 10 published studies. This framework revealed three broad patterns. First, zero-shot evaluations exposed systematic limits to model predictivity that were largely hidden by standard within-dataset cross-validation. These limits were structured varying across brain regions and model classes and depending strongly on the similarity between the images used to build and test the models. Second, current models reproduced some, but not all, classical findings from cognitive neuroscience, providing a diagnosis of the computations that current models have yet to capture. Finally, models that performed well on prediction tests also tended to succeed at reproducing cognitive neuroscience findings, suggesting that prediction and explanation are closely linked. Unlike prediction scores however, cognitive tests help diagnose where and why models fail. Together, the zero-shot evaluation framework provides a scalable and unified approach for evaluating brain models across prediction and cognitive neuroscience tests, which can help us better understand what current models capture and what still remains to be explained.

neuroscience↗

bioRxiv2020

Reconstructing feedback representations in ventral visual pathway with a generative adversarial autoencoder

While vision evokes a dense network of feedforward and feedback neural processes in the brain, visual processes are primarily modeled with feedforward hierarchical neural networks, leaving the computational role of feedback processes poorly understood. Here, we developed a generative autoencoder neural network model and adversarially trained it on a categorically diverse data set of images. We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. We compared representational similarity of the activity patterns in the proposed model with temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) visual brain responses. The proposed generative model identified two segregated neural dynamics in the visual brain. A temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally later dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Our results append to previous studies on neural feedback processes by presenting a new insight into the algorithmic function and the information carried by the feedback processes in the ventral visual pathway. Author summaryIt has been shown that the ventral visual cortex consists of a dense network of regions with feedforward and feedback connections. The feedforward path processes visual inputs along a hierarchy of cortical areas that starts in early visual cortex (an area tuned to low level features e.g. edges/corners) and ends in inferior temporal cortex (an area that responds to higher level categorical contents e.g. faces/objects). Alternatively, the feedback connections modulate neuronal responses in this hierarchy by broadcasting information from higher to lower areas. In recent years, deep neural network models which are trained on object recognition tasks achieved human-level performance and showed similar activation patterns to the visual brain. In this work, we developed a generative neural network model that consists of encoding and decoding sub-networks. By comparing this computational model with the human brain temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) response patterns, we found that the encoder processes resemble the brain feedforward processing dynamics and the decoder shares similarity with the brain feedback processing dynamics. These results provide an algorithmic insight into the spatiotemporal dynamics of feedforward and feedback processes in biological vision.

neuroscience↗