Language-aligned models and structured scene descriptions reveal sensitivity to compositional scene structure in the high-level visual cortex

Abstract

Natural scenes are defined not only by their contained objects, but also by the structured relations among those objects. Using 7T fMRI data from the Natural Scenes Dataset, we tested whether the high-level visual cortex is sensitive to this compositional scene structure. We related narrative scene descriptions to cortical responses in an encoding framework, contrasting intact narrative descriptions with lexical control descriptions that preserved word content while disrupting compositional structure via word randomization. Intact narrative descriptions better predicted cortical responses in high-level visual cortex, indicating sensitivity to structured scene information beyond lexical content alone. This advantage was particularly pronounced for scenes with richer compositional structure. Further, a language-aligned vision model outperformed a self-supervised vision-only model, and this advantage showed a similar high-level cortical distribution. These findings suggest that the high-level visual cortex represents structured scene information beyond simple entity co-occurrence and language supervision may help induce similar representations in artificial vision models.