Biology subjects

Cappaert, N.

Publications and source records attributed to Cappaert, N..

4 recordsLinked to original sources

bioRxiv2025

Shared texture-like representations, not global form, underlie deep neural network alignment with human visual processing

Deep neural networks (DNNs) excel at predicting neural responses across the visual hierarchy 1-5, a success widely interpreted as evidence of shared object recognition computations 6,7. Yet improving DNN object recognition accuracy does not reliably increase neural predictivity 8,9, and even untrained networks predict brain responses above chance 9-11. This disconnect suggests that object recognition may not drive DNN-brain alignment. Texture-like statistics are represented in both DNNs and mid-level visual cortex 12-17. In natural images, these statistics are carried by objects and backgrounds, shaping representations and recognition in both systems 18-21. Does DNN-brain alignment reflect shared sensitivity to object-related information or texture-like statistics? To dissociate these factors, we recorded EEG from 57 participants viewing natural scenes, texture-synthesized images preserving local statistics while disrupting global form and object-only images with backgrounds removed. If alignment reflects texture-like statistics, it should peak for texture-synthesized images. If it reflects object-related processing, alignment should be strongest for natural and object-only conditions, which preserve object information. We compared EEG responses with DNN activations via weighted representational similarity analysis 22,23. Texture-synthesized images yielded the strongest DNN-EEG alignment, peaking in early responses (<200 ms) and explaining up to [~]85% of noise-ceiling-normalized explainable variance versus [~]44% for natural and [~]55% for isolated objects. Crucially, object categories were more decodable for natural and object-only images than texture-synthesized images, yet these object-rich conditions showed weaker alignment. This dissociation reveals that DNNs capture the texture-statistical component of early visual responses while failing to explain later, object-related variance. HighlightsO_LITexture-synthesized images yield the strongest DNN-brain alignment across DNNs. C_LIO_LIObject categories decode better from object-rich than texture-synthesized EEG. C_LIO_LIDNN-brain alignment reflects shared texture statistics, not object-related processing. C_LI

neuroscience↗

bioRxiv2025

Scene segmentation processes drive EEG-DCNN alignment

Visual processing in biological and artificial neural networks has been extensively studied through the lens of object recognition. While deep convolutional neural networks (DCNNs) have demonstrated hierarchical feature extraction similar to biological systems (DiCarlo and Cox, 2007; Yamins and DiCarlo, 2016), recent findings reveal a growing discrepancy: DCNNs with higher object categorization accuracy paradoxically show worse performance at predicting neural responses (Xu and Vaziri-Pashkam, 2021; Linsley et al., 2023). Using a large-scale human electroencephalography (EEG) dataset (n=10, 82,160 trials), we investigate whether this discrepancy arises because human neural EEG signals predominantly reflect scene segmentation processes rather than high-level, category-specific object representations. We trained DCNNs to perform object recognition using visual diets ([~]1 million training images across 292 object categories) with systematically varying scene segmentation demands: objects-only (pre-segmented), background-silhouette (explicit boundaries), original/background-only images (requiring full segmentation). Despite substantial differences in categorization accuracy (27-53%), all trained models showed remarkably uniform encoding performance, with peak correlations with neural data at [~]0.1s post-stimulus. Layer-wise analysis revealed a significant negative correlation between categorization accuracy and encoding performance, with earlier network layers better predicting EEG responses than deeper layers specialized for object categorization. This dissociation suggests that EEG signals primarily reflect fundamental scene parsing mechanisms rather than object-specific representations, explaining the growing discrepancy between DCNNs increasing categorization performance but deteriorating neural prediction performance. Significance StatementThis research provides a novel perspective on human electroencephalography (EEG) signals during visual processing through systematic manipulation of scene segmentation demands in deep neural networks. Using a large-scale dataset of 82,160 EEG trials and 20 trained DCNNs, we demonstrate that EEG responses primarily reflect early visual processing involved in breaking down and organizing visual scenes (scene segmentation/parsing) rather than high-level object recognition. This finding helps explain previously observed discrepancies between DCNNs categorization performance and neural prediction accuracy, suggesting that improving models ability to segment scenes, rather than simply recognizing isolated objects, may better align artificial and biological visual processing.

neuroscience↗

bioRxiv2023

Human visual cortex and deep convolutional neural network care deeply about object background

Deep convolutional neural networks (DCNNs) are able to predict brain activity during object categorization tasks, but factors contributing to this predictive power are not fully understood. Our study aimed to investigate the factors contributing to the predictive power of DCNNs in object categorization tasks. We compared the activity of four DCNN architectures with electroencephalography (EEG) recordings obtained from 62 human subjects during an object categorization task. Previous physiological studies on object categorization have highlighted the importance of figure-ground segregation - the ability to distinguish objects from their backgrounds. Therefore, we set out to investigate if figure-ground segregation could explain DCNNs predictive power. Using a stimuli set consisting of identical target objects embedded in different backgrounds, we examined the influence of object background versus object category on both EEG and DCNN activity. Crucially, the recombination of naturalistic objects and experimentally-controlled backgrounds creates a sufficiently challenging and naturalistic task, while allowing us to retain experimental control. Our results showed that early EEG activity (<100ms) and early DCNN layers represent object background rather than object category. We also found that the predictive power of DCNNs on EEG activity is related to processing of object backgrounds, rather than categories. We provided evidence from both trained and untrained (i.e. random weights) DCNNs, showing figure-ground segregation to be a crucial step prior to the learning of object features. These findings suggest that both human visual cortex and DCNNs rely on the segregation of object backgrounds and target objects in order to perform object categorization. Altogether, our study provides new insights into the mechanisms underlying object categorization as we demonstrated that both human visual cortex and DCNNs care deeply about object background. Author summaryOur study aimed to investigate the factors contributing to the predictive power of deep convolutional neural networks (DCNNs) on EEG activity in object recognition tasks. We compared the activity of four DCNN architectures with human neural recordings during an object categorization task. We used a stimuli set consisting of identical target objects embedded in different phase-scrambled backgrounds. The distinction between object backgrounds and object categories allows us to investigate the influence of either factor for human subjects and DCNNs. Surprisingly, we found that both human visual processing and early DCNNs layers dedicate a large proportion of activity to processing object backgrounds instead of object category. Furthermore, this shared ability to make object backgrounds (and not just object category) invariant is largely the reason why DCNNs are predictive of brain dynamics in our experiment. We posit this shared ability to be an important solution for object categorization. Finally, we conclude that DCNNs, like humans, care deeply about object backgrounds.

neuroscience↗

bioRxiv2022

A critical test of deep convolutional neural networks ability to capture recurrent processing in the brain using visual masking

Recurrent processing is a crucial feature in human visual processing supporting perceptual grouping, figure-ground segmentation, and recognition under challenging conditions. There is a clear need to incorporate recurrent processing in deep convolutional neural networks (DCNNs) but the computations underlying recurrent processing remain unclear. In this paper, we tested a form of recurrence in deep residual networks (ResNets) to capture recurrent processing signals in the human brain. Though ResNets are feedforward networks, they approximate an excitatory additive form of recurrence. Essentially, this form of recurrence consists of repeating excitatory activations in response to a static stimulus. Here, we used ResNets of varying depths (reflecting varying levels of recurrent processing) to explain electroencephalography (EEG) activity within a visual masking paradigm. Sixty-two humans and fifty artificial agents (10 ResNet models of depths - 4, 6, 10, 18 and 34) completed an object categorization task. We show that deeper networks (ResNet-10, 18 and 34) explained more variance in brain activity compared to shallower networks (ResNet-4 and 6). Furthermore, all ResNets captured differences in brain activity between unmasked and masked trials, with differences starting at [~]98ms (from stimulus onset). These early differences indicated that EEG activity reflected pure feedforward signals only briefly (up to [~]98ms). After [~]98ms, deeper networks showed a significant increase in explained variance which peaks at [~]200ms, but only within unmasked trials, not masked trials. In summary, we provided clear evidence that excitatory additive recurrent processing in ResNets captures some of the recurrent processing in humans. Significance statementThe challenge of modeling recurrent processes is not trivial and the operationalization of recurrent processing is highly contested. In this paper, we tested the ability of deep residual networks (ResNets) to explain recurrent processes in the human brain. Though ResNets are feedforward networks, they have been shown to equate operations in recurrent neural networks. In this study, we show that deeper networks explained more variance in brain activity than shallower networks. However, all networks still performed far from the noise ceiling. Thus, we conclude that recurrent processing in ResNets captures a form of recurrent processing in humans though other types of recurrent processing (inhibition, multiplicative) that are not present in current regular deep neural networks (alexnet, cornet, resnet) are necessary for building better visual models.

neuroscience↗