Biology subjects

Cichy, R. M.

Publications and source records attributed to Cichy, R. M..

At least 37 records · Page 2Linked to original sources

bioRxiv2023

Coherent categorical information triggers integration-related alpha dynamics

To create coherent visual experiences, the brain spatially integrates the complex and dynamic information it receives from the environment. We previously demonstrated that feedback-related alpha activity carries stimulus-specific information when two spatially and temporally coherent naturalistic inputs can be integrated into a unified percept. In this study, we sought to determine whether such integration-related alpha dynamics are triggered by categorical coherence in visual inputs. In an EEG experiment, we manipulated the degree of coherence by presenting pairs of videos from the same or different categories through two apertures in the left and right visual hemifields. Critically, video pairs could be video-level coherent (i.e., stem from the same video), coherent in their basic-level category, coherent in their superordinate category, or incoherent (i.e., stem from videos from two entirely different categories). We conducted multivariate classification analyses on rhythmic EEG responses to decode between the video stimuli in each condition. As the key result, we significantly decoded the video-level coherent and basic-level coherent stimuli, but not the superordinate coherent and incoherent stimuli, from cortical alpha rhythms. This suggests that alpha dynamics play a critical role in integrating information across space, and that cortical integration processes are flexible enough to accommodate information from different exemplars of the same basic-level category.

neuroscience↗

bioRxiv2023

The link between visual representations and behavior in human scene perception

1.Scene recognition is a core sensory capacity that enables humans to adaptively interact with their environment. Despite substantial progress in the understanding of the neural representations underlying scene recognition, the relevance of these representations for behavior given varying task demands remains unknown. To address this, we aimed to identify behaviorally relevant scene representations, to characterize them in terms of their underlying visual features, and to reveal how they vary across different tasks. We recorded fMRI data while human participants viewed scenes and linked brain responses to behavior in three tasks acquired in separate sessions: manmade/natural categorization, basic-level categorization, and fixation color discrimination. We found correlations between categorization response times and scene-specific brain responses, quantified as the distance to a hyperplane derived from a multivariate classifier. Across tasks, these effects were found in largely distinct parts of the ventral visual stream. This suggests that different scene representations are relevant for behavior depending on the task. Next, using deep neural networks as a proxy for visual feature representations, we found that early/intermediate layers mediated the relationship between scene representations and behavior for both categorization tasks, indicating a contribution of low-/mid-level visual features to these representations. Finally, we observed opposite patterns of brain-behavior correlations in the manmade/natural and the fixation task, indicating interference of representations with behavior for task demands that do not align with the content of representations. Together, these results reveal the spatial extent, content, and task-dependence of the visual representations that mediate behavior in complex scenes.

neuroscience↗

bioRxiv2023

An adversarial collaboration to critically evaluate theories of consciousness

Different theories explain how subjective experience arises from brain activity1,2. These theories have independently accrued evidence, yet, confirmation bias and dependence on design choices hamper progress in the field3. Here, we present an open science adversarial collaboration which directly juxtaposes Integrated Information Theory (IIT)4,5 and Global Neuronal Workspace Theory (GNWT)6-10, employing a theory-neutral consortium approach11,12. We investigate neural correlates of the content and duration of visual experience. The theory proponents and the consortium developed and preregistered the experimental design, divergent predictions, expected outcomes, and their interpretation12. 256 human subjects viewed suprathreshold stimuli for variable durations while neural activity was measured with functional magnetic resonance imaging, magnetoencephalography, and electrocorticography. We find information about conscious content in visual, ventro-temporal and inferior frontal cortex, with sustained responses in occipital and lateral temporal cortex reflecting stimulus duration, and content-specific synchronization between frontal and early visual areas. These results confirm some predictions of IIT and GNWT, while substantially challenging both theories: for IIT, a lack of sustained synchronization within posterior cortex contradicts the claim that network connectivity specifies consciousness. GNWT is challenged by the general lack of ignition at stimulus offset and limited representation of certain conscious dimensions in prefrontal cortex. Beyond challenging the theories themselves, we present an alternative approach to advance cognitive neuroscience through a principled, theory-driven, collaborative effort. We highlight the challenges to change peoples mind 13 and the need for a quantitative framework integrating evidence for systematic theory testing and building.

neuroscience↗

bioRxiv2023

Attention Modulates Human Visual Responses to Objects by Tuning Sharpening

Visual stimuli compete with each other for cortical processing and attention biases this competition in favor of the attended stimulus. How does the relationship between the stimuli affect the strength of this attentional bias? Here, we used functional MRI to explore the effect of target-distractor similarity in neural representation on attentional modulation in the human visual cortex using univariate and multivariate pattern analyses. Using stimuli from four object categories (human bodies, cats, cars and houses), we investigated attentional effects in the primary visual area V1, the object-selective regions LO and pFs, the body-selective region EBA, and the scene-selective region PPA. We demonstrated that the strength of the attentional bias towards the target is not fixed but decreases with increasing target-distractor similarity. Simulations provided evidence that this result pattern is explained by tuning sharpening rather than an increase in gain. Our findings provide a mechanistic explanation for behavioral effects of target-distractor similarity on attentional biases and suggest tuning sharpening as the underlying mechanism in object-based attention.

neuroscience↗

bioRxiv2023

The influence of the bullseye versus standard fixation cross on eye movements and classifying natural images from EEG

Withdrawn StatementThe authors have withdrawn this manuscript because we detected fundamental errors in the formula we used to convert eye tracking data (x and y coordinates) from pixels to degrees. After addressing these errors, the results changed considerably. On the one hand, the behavioral results point towards differences for the bullseye versus the standard fixation cross different and larger than originally reported. However, the eyetracking decoding difference between standard and bullseye fixation cross is no longer significant. Therefore, the main claim of the paper - that bullseye fixation cross leads to lower eye movements compared to standard fixation cross, in turn resulting in lower decoding performance of object identity for eyetracking and EEG data - no longer stands. Given this change in the main results, we therefore withdraw the preprint.. Therefore, the authors do not wish this work to be cited as reference for the project. If you have any questions, please contact the corresponding author.

neuroscience↗

bioRxiv2023

Alpha-frequency feedback to early visual cortex orchestrates coherent natural vision

During natural vision, the brain generates coherent percepts by integrating sensory inputs scattered across the visual field. Here, we asked whether this integration process is mediated by rhythmic cortical feedback. In EEG and fMRI experiments, we experimentally manipulated the demand for integration by changing the spatiotemporal coherence of natural videos presented across visual hemifields. Our EEG data revealed that information about incongruent videos is coded in feedforward-related gamma activity while information about congruent videos is coded in feedback-related alpha activity, indicating that integration is indeed mediated by rhythmic feedback. Our fMRI data identified scene-selective cortex as a likely source of this feedback. Analytically combining our EEG and fMRI data further revealed that feedback-related representations in the alpha band shape the earliest stages of visual processing in cortex. Together, our findings indicate that the construction of coherent visual experiences relies on rhythmic cortical feedback that fully traverses the visual hierarchy.

neuroscience↗

bioRxiv2023

Empirically identifying and computationally modelling the brain-behaviour relationship for human scene categorization

Humans effortlessly make quick and accurate perceptual decisions about the nature of their immediate visual environment, such as the category of the scene they face. Previous research has revealed a rich set of cortical representations potentially underlying this feat. However, it remains unknown which of these representations are suitably formatted for decision-making. Here, we approached this question empirically and computationally, using neuroimaging and computational modelling. For the empirical part, we collected electroencephalography (EEG) data and reaction times from human participants during a scene categorization task (natural vs. man-made). We then related neural representations to behaviour using a multivariate extension of signal detection theory. We observed a correlation specifically between [~]100 ms and [~]200 ms after stimulus onset, suggesting that the neural scene representations in this time period are suitably formatted for decision-making. For the computational part, we evaluated a recurrent convolutional neural network (RCNN) as a model of brain and behaviour. Unifying our previous observations in an image-computable model, the RCNN predicted well the neural representations, the behavioural scene categorization data, as well as the relationship between them. Our results identify and computationally characterize the neural and behavioural correlates of scene categorization in humans. Significance statementCategorizing scene information is a ubiquitous and crucial task. Here we provide an empirical and computational account of scene categorization. Previous research has identified when scenes are represented in the visual processing hierarchy, but it remains unclear which of these representations are relevant for behaviour. We identified such representations between [~]100 ms and [~]200 ms after stimulus onset. We then showed that scene categorization in humans can be modelled via a recurrent convolutional neural network in a unified manner, i.e., in terms of neural and behavioural correlates, and their relationship. Together this reveals which representations underlie scene categorization behaviour and proposes a computational mechanism that implements such representations.

neuroscience↗

bioRxiv2022

The spatiotemporal neural dynamics of object recognition for natural images and line drawings

1.Drawings offer a simple and efficient way to communicate meaning. While line drawings capture only coarsely how objects look in reality, we still perceive them as resembling real-world objects. Previous work has shown that this perceived similarity is mirrored by shared neural representations for drawings and natural images, which suggests that similar mechanisms underlie the recognition of both. However, other work has proposed that representations of drawings and natural images become similar only after substantial processing has taken place, suggesting distinct mechanisms. To arbitrate between those alternatives, we measured brain responses resolved in space and time using fMRI and MEG, respectively, while human participants (female and male) viewed images of objects depicted as photographs, line drawings, or sketch-like drawings. Using multivariate decoding, we demonstrate that object category information emerged similarly fast and across overlapping regions in occipital, ventral-temporal and posterior parietal cortex for all types of depiction, yet with smaller effects at higher levels of visual abstraction. In addition, cross-decoding between depiction types revealed strong generalization of object category information from early processing stages on. Finally, by combining fMRI and MEG data using representational similarity analysis, we found that visual information traversed similar processing stages for all types of depiction, yet with an overall stronger representation for photographs. Together our results demonstrate broad commonalities in the neural dynamics of object recognition across types of depiction, thus providing clear evidence for shared neural mechanisms underlying recognition of natural object images and abstract drawings. 2. Significance StatementWhen we see a line drawing, we effortlessly recognize it as an object in the world despite its simple and abstract style. Here we asked to what extent this correspondence in perception is reflected in the brain. To answer this question, we measured how neural processing of objects depicted as photographs and line drawings with varying levels of detail (from natural images to abstract line drawings) evolves over space and time. We find broad commonalities in the spatiotemporal dynamics and the neural representations underlying the perception of photographs and even abstract drawings. These results indicate a shared basic mechanism supporting recognition of drawings and natural images.

neuroscience↗

bioRxiv2022

A Multivariate Comparison of EEG and fMRI to ECoG Using Visual Object Representations in Humans

1.Today most neurocognitive studies in humans employ the non-invasive neuroimaging techniques fMRI and EEG. However, how the data provided by fMRI and EEG relate exactly to the underlying neural activity remains incompletely understood. Here, we pursued this open question by relating EEG and fMRI data at the level of neural population codes using multivariate pattern analysis. For this, we recorded EEG and fMRI data while participants viewed everyday objects in different viewing conditions, and then related the data to ECoG data recorded for the same stimulus set. The comparison of EEG and ECoG data showed that object category signals emerge swiftly in the visual system and can be detected by both EEG and ECoG at similar temporal delays after stimulus onset. The correlation between EEG and ECoG reduces and onset latencies increase when object representations tolerant to changes in scale and orientation are considered. The comparison of fMRI and ECoG overall revealed a tighter relationship in occipital than in temporal regions, related to differences in fMRI signal-to-noise ratio. Together, our results reveal a complex relationship between fMRI, EEG and ECoG signals at the level of population codes that critically depends on the time point after stimulus onset, the region investigated, and the visual contents used.

neuroscience↗

bioRxiv2022

Independent spatiotemporal effects of spatial attention and background clutter on human object location representations

1.Spatial attention helps us to efficiently localize objects in cluttered environments. However, the processing stage at which spatial attention modulates object location representations remains unclear. Here we investigated this question identifying processing stages in time and space in an EEG and fMRI experiment respectively. As both object location representations and attentional effects have been shown to depend on the background on which objects appear, we included object background as an experimental factor. During the experiments, human participants viewed images of objects appearing in different locations on blank or cluttered backgrounds while either performing a task on fixation or on the periphery to direct their covert spatial attention away or towards the objects. We used multivariate classification to assess object location information. Consistent across the EEG and fMRI experiment, we show that spatial attention modulated location representations during late processing stages (>150ms, in middle and high ventral visual stream areas) independent of background condition. Our results clarify the processing stage at which attention modulates object location representations in the ventral visual stream and show that attentional modulation is a cognitive process separate from recurrent processes related to the processing of objects on cluttered backgrounds.

neuroscience↗

bioRxiv2022

A large and rich EEG dataset for modeling human visual object recognition

The human brain achieves visual object recognition through multiple stages of nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output M/EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.

neuroscience↗

bioRxiv2021

Benchmarking GE-BOLD, SE-BOLD, and SS-SI-VASO sequences for depth-dependent separation of feedforward and feedback signals in high-field MRI

Recent advances in high-field fMRI have allowed differentiating feedforward and feedback information in the grey matter of the human brain. For continued progress in this endeavor, it is critical to understand how MRI data acquisition parameters impact the read-out of information from laminar response profiles. Here, we benchmarked three different MR-sequences at 7T - gradient-echo (GE), spin-echo (SE) and vascular space occupancy imaging (VASO) - in differentiating feedforward and feedback signals in human early visual cortex (V1). The experiment (N=4) consisted of two complementary tasks: a perception task that predominantly evokes feedforward signals and a working memory task that relies on feedback signals. In the perception task, participants saw flickering oriented gratings while detecting orthogonal color-changes. In the working memory task, participants memorized the precise orientation of a grating. We used multivariate pattern analysis to read out the perceived (feedforward) and memorized (feedback) grating orientation from neural signals across cortical depth. Analyses across all the MR-sequences revealed perception signals predominantly in the middle cortical compartment of area V1 and working memory signals in the deep compartment. Despite an overall consistency across sequences, SE-EPI was the only sequence where both feedforward and feedback information were differently pronounced across cortical depth in a statistically robust way. We therefore suggest that in the context of a typical cognitive neuroscience experiment as the one benchmarked here, SE-EPI may provide a favorable trade-off between spatial specificity and signal sensitivity. HighlightsHere, we benchmarked three sequences at high-field fMRI -GE-BOLD, SE-BOLD and VASO - in differentiating feedforward and feedback signals across grey matter depth of area V1. We show that: O_LIAll the MR-sequences revealed the feedforward and feedback signals at the middle and deep cortical bins, respectively. C_LIO_LISuch correspondence across the sequences indicates that widely used GE-BOLD is a suitable method for the exploration of signals in cortical depth. C_LIO_LIOnly SE-BOLD yielded statistically reliable differences between the cortical bins carry- ing feedforward and feedback signals. C_LI

neuroscience↗

bioRxiv2021

Resolving the time course of visual and auditory object categorization

Humans can effortlessly categorize objects, both when they are conveyed through visual images and spoken words. To resolve the neural correlates of object categorization, studies have so far primarily focused on the visual modality. It is therefore still unclear how the brain extracts categorical information from auditory signals. In the current study we used EEG (N=47) and time-resolved multivariate pattern analysis to investigate (1) the time course with which object category information emerges in the auditory modality and (2) how the representational transition from individual object identification to category representation compares between the auditory modality and the visual modality. Our results show that (1) that auditory object category representations can be reliably extracted from EEG signals and (2) a similar representational transition occurs in the visual and auditory modalities, where an initial representation at the individual-object level is followed by a subsequent representation of the objects category membership. Altogether, our results suggest an analogous hierarchy of information processing across sensory channels. However, we did not find evidence for a shared supra-modal code, suggesting that the contents of the different sensory hierarchies are ultimately modality-unique.

animal behavior and cognition↗

bioRxiv2021

Visual category representations in the infant brain

Visual categorization is a human core cognitive capacity 1,2 that depends on the development of visual category representations in the infant brain 3-7. However, the exact nature of infant visual category representations and their relationship to the corresponding adult form remains unknown 8. Our results clarify the nature of visual category representations from electroencephalography (EEG) data in 6- to 8-month-old infants and their developmental trajectory towards adult maturity in the key characteristics of temporal dynamics 2,9, representational format 10-12, and spectral properties 13,14. Temporal dynamics change from slowly emerging, developing representations in infants to quickly emerging, complex representations in adults. Despite those differences, infants and adults already partly share visual category representations. The format of infants representations is visual features of low to intermediate complexity, whereas adults representations also encode high complexity features. Theta band activity contributes to visual category representations in infants, and these representations are shifted to the alpha/beta band in adults. Together, we reveal the developmental neural basis of visual categorization in humans, show how information transmission channels change in development, and demonstrate the power of advanced multivariate analysis techniques in infant EEG research for theory building in developmental cognitive science.

neuroscience↗

bioRxiv2021

Deep neural networks and visuo-semantic models explain complementary components of human ventral-stream representational dynamics

Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural timeseries data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography (MEG) data acquired in human participants (9 females, 6 males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object rep-resentations unfold over space and time. While lower-level visual areas are better explained by DNN features, especially during the early phase of the response (< 128 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features during a later time window (starting 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured, and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral stream computations. SIGNIFICANCE STATEMENTWhen we view objects such as faces and cars in our visual environment, their neural representations dynamically unfold over time at a millisecond scale. These dynamics reflect the cortical computations that support fast and robust object recognition. Deep neural networks (DNNs) have emerged as a promising framework for modeling these computations but cannot yet fully account for the neural dynamics. Using magnetoencephalography data acquired in human observers during object viewing, we show that readily nameable aspects of objects, such as "eye", "wheel", and "face", can account for variance in the neural dynamics over and above DNNs. These findings suggest that DNNs and humans may in part rely on different object features for visual recognition and provide guidelines for model improvement.

neuroscience↗

bioRxiv2021

Disentangling dimensions of animacy in human brain and behaviour

Distinguishing animate from inanimate things is of great behavioural importance. Despite distinct brain and behavioural responses to animate and inanimate things, it remains unclear which object properties drive these responses. Here, we investigate the importance of five object dimensions related to animacy ("being alive", "looking like an animal", "having agency", "having mobility", and "being unpredictable") in brain (fMRI, EEG) and behaviour (property and similarity judgements) of 19 participants. We used a stimulus set of 128 images, optimized by a genetic algorithm to disentangle these five dimensions. The five dimensions explained much variance in the similarity judgments. Each dimension also explained a modest but significant amount of variance in the brain representations, except, surprisingly, "being alive". Different brain regions sensitive to animacy may represent distinct dimensions, either as accessible perceptual stepping stones toward detecting whether something is alive or because they are of behavioural importance in their own right.

neuroscience↗

bioRxiv2021

Semantic scene-object consistency modulates N300/400 EEG components, but does not automatically facilitate object representations

During natural vision, objects rarely appear in isolation, but often within a semantically related scene context. Previous studies reported that semantic consistency between objects and scenes facilitates object perception, and that scene-object consistency is reflected in changes in the N300 and N400 components in EEG recordings. Here, we investigate whether these N300/400 differences are indicative of changes in the cortical representation of objects. In two experiments, we recorded EEG signals while participants viewed semantically consistent or inconsistent objects within a scene; in Experiment 1, these objects were task-irrelevant, while in Experiment 2, they were directly relevant for behavior. In both experiments, we found reliable and comparable N300/400 differences between consistent and inconsistent scene-object combinations. To probe the quality of object representations, we performed multivariate classification analyses, in which we decoded the category of the objects contained in the scene. In Experiment 1, in which the objects were not task-relevant, object category could be decoded from around 100 ms after the object presentation, but no difference in decoding performance was found between consistent and inconsistent objects. By contrast, when the objects were task-relevant in Experiment 2, we found enhanced decoding of semantically consistent, compared to semantically inconsistent, objects. These results show that differences in N300/400 components related to scene-object consistency do not index changes in cortical object representations, but rather reflect a generic marker of semantic violations. Further, our findings suggest that facilitatory effects between objects and scenes are task-dependent rather than automatic.

neuroscience↗

bioRxiv2021

Modelling brain representations of abstract concepts

Abstract conceptual representations are critical for human cognition. Despite their importance, key properties of these representations remain poorly understood. Here, we used computational models of distributional semantics to predict multivariate fMRI activity patterns during the activation and contextualization of abstract concepts. We devised a task in which participants had to embed abstract nouns into a story that they developed around a given background context. We found that representations in inferior parietal cortex were predicted by concept similarities emerging in models of distributional semantics. By constructing different model families, we reveal the models learning trajectories and delineate how abstract and concrete training materials contribute to the formation of brain-like representations. These results inform theories about the format and emergence of abstract conceptual representations in the human brain.

neuroscience↗