Biology subjects

Isik, L.

Publications and source records attributed to Isik, L..

13 recordsLinked to original sources

bioRxiv2026

Multidimensional dynamics of object representations in the human visual system

AO_SCPLOWBSTRACTC_SCPLOWNatural image representations are distributed across many dimensions of visual cortex activity, but little is known about how the multidimensional structure of these representations evolves over time following stimulus onset. Here we examined the temporal dynamics and latent dimensional structure of natural object representations in large-scale EEG and MEG data. We also compared these data with leading representational models derived from large-scale human similarity judgments and deep neural networks. Our findings reveal a rapid expansion of stimulus dimensionality in the brain, which peaks within 100 milliseconds and gradually decays over hundreds of milliseconds. The dynamics of these dimensionality changes tracked the decoding accuracy for both behavioral embeddings and neural network features, suggesting that dimensionality may be a general indicator of representational expressivity. Interestingly, the dimensionality of the neural representations could not be fully explained by leading behavior-based or neural network models. Follow-up experiments showed that the remaining neural variance carried additional perceptually relevant information not yet explained by leading models. Together, these findings reveal previously unrecognized complexity in measurements of dynamic human brain responses to natural objects.

neuroscience↗

bioRxiv2026

Bottom-up and generative computations both uniquely explain social brain responses

Making social evaluations from visual input is a core human ability that engages brain regions involved in social perception, including portions of the superior temporal sulcus (STS), as well as higher-level mentalizing regions, such as the temporoparietal junction (TPJ). One common hypothesis proposes that these regions operate hierarchically: social perception regions like posterior STS (pSTS) implement bottom-up computations to generate fast, stimulus-derived representations of social interactions, while mentalizing regions like TPJ perform inverse-planning computations to infer the underlying goals and motivations driving agents behavior. However, this computational-neural mapping has never been formally tested, in large part due to the lack of successful computational models of social processing. We developed computational models aligned with these two frameworks: a graph-neural-network (GNN) model that recognizes social interactions by relying on relational visual information, and a generative inverse-planning model that does so by inverting a model of agents goals and the physical world. In this preregistered study, we collected fMRI responses while participants watched videos of agentive animated shapes depicting social interactions and compared neural responses to both computational models. Surprisingly, we found that both the GNN and inverse-planning model explained neural responses in pSTS and TPJ, even after controlling for variance explained by the other model. Exploratory analyses, however, revealed a shift from early perceptual processing towards later higher-order reasoning in both regions, suggesting a temporal rather than spatial hierarchy. Overall, this study provides the first evidence that both social perception and mentalizing regions carry out a combination of relational bottom-up and higher-level inferential computations, perhaps on distinct timescales. This work also provides the first comparison of an inverse-planning model to neural activity and demonstrates that theory-driven cognitive models can successfully predict fMRI responses to social scenes. Significance StatementThe ability to recognize social interactions between others is central to humans daily lives and engages brain regions supporting social perception and mental state inference. The neural computations underlying this ability, however, are poorly understood. Here we leveraged new models of bottom-up social perception and generative social inference to test the hypothesis that these complementary computations are carried out in separate brain regions. We compared both models to brain responses from subjects viewing procedurally generated videos of social interactions. Surprisingly, we found that both models explained neural activity in both perceptual and mentalizing regions, even when controlling for effects of the other model. These findings challenge the idea of a strict division of labor in the social brain and refine our understanding of the computations supporting human social inference.

animal behavior and cognition↗

bioRxiv2025

Monkey See, Model Knew: Large Language Models Accurately Predict Visual Brain Responses in Humans and Non-Human Primates

AO_SCPLOWBSTRACTC_SCPLOWRecent progress in multimodal AI and language-aligned visual representation learning has rekindled debates about the role of language in shaping the human visual system. In particular, the emergent ability of language-aligned vision models (e.g. CLIP) - and even pure language models (e.g. BERT) - to predict image-evoked brain activity has led some to suggest that human visual cortex itself may be language-aligned in comparable ways. But what would we make of this claim if the same procedures could model visual activity in a species without language? Here, we conducted controlled comparisons of pure-vision, pure-language, and multimodal vision-language models in their prediction of human (N=4) and rhesus macaque (N=6, 5:IT, 1:V1) ventral visual activity to the same set of 1000 captioned natural images (the NSD1000). The results revealed markedly similar patterns in model predictivity of early and late ventral visual cortex across both species. This suggests that language model predictivity of the human visual system is not necessarily due to the evolution or learning of language perse, but rather to the statistical structure of the visual world that is reflected in natural language.

neuroscience↗

bioRxiv2023

Multidimensional neural representations of social features during movie viewing

The social world is dynamic and contextually embedded. Yet, most studies utilize simple stimuli that do not capture the complexity of everyday social episodes. To address this, we implemented a movie viewing paradigm and investigated how the everyday social episodes are processed in the brain. Participants watched one of two movies during an MRI scan. Neural patterns from brain regions involved in social perception, mentalization, action observation, and sensory processing were extracted. Representational similarity analysis results revealed that several labeled social features (including social interaction, mentalization, the actions of others, characters talking about themselves, talking about others, and talking about objects) were represented in superior temporal gyrus (STG) and middle temporal gyrus (MTG). The mentalization feature was also represented throughout the theory of mind network, and characters talking about others engaged the temporoparietal junction (TPJ), suggesting that listeners may spontaneously infer the mental state of those being talked about. In contrast, we did not observe the action representations in frontoparietal regions of the action observation network. The current findings indicate that STG and MTG serve as central hubs for social processing, and that listening to characters talk about others elicits spontaneous mental state inference in TPJ during natural movie viewing.

neuroscience↗

bioRxiv2023

Rapid processing of observed touch through social perceptual brain regions: an EEG-fMRI fusion study

Seeing social touch triggers a strong social-affective response that involves multiple brain networks, including visual, social perceptual, and somatosensory systems. Previous studies have identified the specific functional role of each system, but little is known about the speed and directionality of the information flow. Is this information extracted via the social perceptual system or from simulation from somatosensory cortex? To address this, we examined the spatiotemporal neural processing of observed touch. Twenty participants watched 500 ms video clips showing social and non-social touch during EEG recording. Visual and social-affective features were rapidly extracted in the brain, beginning at 90 and 150 ms after video onset, respectively. Combining the EEG data with fMRI data from our prior study with the same stimuli reveals that neural information first arises in early visual cortex (EVC), then in the temporoparietal junction and posterior superior temporal sulcus (TPJ/pSTS), and finally in the somatosensory cortex. EVC and TPJ/pSTS uniquely explain EEG neural patterns, while somatosensory cortex does not contribute to EEG patterns alone, suggesting that social-affective information may flow from TPJ/pSTS to somatosensory cortex. Together, these findings show that social touch is processed quickly, within the timeframe of feedforward visual processes, and that the social-affective meaning of touch is first extracted by a social perceptual pathway. Such rapid processing of social touch may be vital to its effective use during social interaction. Significance StatementSeeing physical contact between people evokes a strong social-emotional response. Previous research has identified the brain systems responsible for this response, but little is known about how quickly and in what direction the information flows. We demonstrated that the brain processes the social-emotional meaning of observed touch quickly, starting as early as 150 milliseconds after the stimulus onset. By combining EEG data with fMRI data, we show for the first time that the social-affective meaning of touch is first extracted by a social perceptual pathway and followed by the later involvement of somatosensory simulation. This rapid processing of touch through the social perceptual route may play a pivotal role in effective usage of touch in social communication and interaction.

neuroscience↗

bioRxiv2023

Disentangled deep generative models reveal coding principles of the human face processing network

Despite decades of research, much is still unknown about the computations carried out in the human face processing network. Recently deep networks have been proposed as a computational account of human visual processing, but while they provide a good match to neural data throughout visual cortex, they lack interpretability. We introduce a method for interpreting brain activity using a new class of deep generative models, disentangled representation learning models, which learn a low-dimensional latent space that "disentangles" different semantically meaningful dimensions of faces, such as rotation, lighting, or hairstyle, in an unsupervised manner by enforcing statistical independence between dimensions. We find that the majority of our models learned latent dimensions are interpretable by human raters. Further, these latent dimensions serve as a good encoding model for human fMRI data. We next investigated the representation of different latent dimensions across face-selective voxels. We find a gradient from low- to high-level face feature representations along posterior to anterior face-selective regions, corroborating prior models of human face recognition. Interestingly, though, we find no spatial segregation between identity-relevant and irrelevant face features. Finally, we provide new insight into the few "entangled" (uninterpretable) dimensions in our model by showing that they match responses across the ventral stream and carry significant information about facial identity. Disentangled face encoding models provide an exciting alternative to standard "black box" deep learning approaches for modeling and interpreting human brain data.

neuroscience↗

bioRxiv2022

A shared neural code for social interaction encoding and memory in the human superior temporal sulcus

Recognizing and remembering social information is a crucial cognitive skill. Neural patterns in the superior temporal sulcus (STS) support our ability to perceive others social interactions. However, despite the prominence of social interactions in memory, the neural basis of retrieving social interactions is still unknown. To fill this gap, we investigated the brain mechanisms underlying memory of others social interactions during free spoken recall of a naturalistic movie. By applying machine learning-based fMRI encoding analyses to densely labeled movie and recall data we found that STS activity patterns evoked by viewing social interactions predicted neural responses to social interaction memories. This finding suggests that the STS contains high-level conceptual, representations of social interactions, and its reactivation underlies our ability to remember others interactions.

neuroscience↗

bioRxiv2022

A data-driven investigation of human action representations

Understanding actions performed by others requires us to integrate different types of information about people, scenes, objects, and their interactions. What organizing dimensions does the mind use to make sense of this complex action space? To address this question, we collected intuitive similarity judgments across two large-scale sets of naturalistic videos depicting everyday actions. We used cross-validated sparse non-negative matrix factorization (NMF) to identify the structure underlying action similarity judgments. A low-dimensional representation, consisting of nine to ten dimensions, was sufficient to accurately reconstruct human similarity judgments. The dimensions were robust to stimulus set perturbations and reproducible in a separate odd-one-out experiment. Human labels mapped these dimensions onto semantic axes relating to food, work, and home life; social axes relating to people and emotions; and one visual axis related to scene setting. While highly interpretable, these dimensions did not share a clear one-to-one correspondence with prior hypotheses of action-relevant dimensions. Together, our results reveal a low-dimensional set of robust and interpretable dimensions that organize intuitive action similarity judgments and highlight the importance of data-driven investigations of behavioral representations.

neuroscience↗

bioRxiv2021

Social-affective features drive human representations of observed actions

Humans observe actions performed by others in many different visual and social settings. What features do we extract and attend when we view such complex scenes, and how are they processed in the brain? To answer these questions, we curated two large-scale sets of naturalistic videos of everyday actions and estimated their perceived similarity in two behavioral experiments. We normed and quantified a large range of visual, action-related and social-affective features across the stimulus sets. Using a cross-validated variance partitioning analysis, we found that social-affective features predicted similarity judgments better than, and independently of, visual and action features in both behavioral experiments. Next, we conducted an electroencephalography (EEG) experiment, which revealed a sustained correlation between neural responses to videos and their behavioral similarity. Visual, action, and social-affective features predicted neural patterns at early, intermediate and late stages respectively during this behaviorally relevant time window. Together, these findings show that social-affective features are important for perceiving naturalistic actions, and are extracted at the final stage of a temporal gradient in the brain.

neuroscience↗

bioRxiv2021

Is it that simple? Linear mapping models in cognitive neuroscience

Many cognitive neuroscience studies use large feature sets to predict and interpret brain activity patterns. Feature sets take many forms, from human stimulus annotations to representations in deep neural networks. Of crucial importance in all these studies is the mapping model, which defines the space of possible relationships between features and neural data. Until recently, most encoding and decoding studies have used linear mapping models. Increasing availability of large datasets and computing resources has recently allowed some researchers to employ more flexible nonlinear mapping models instead; however, the question of whether nonlinear mapping models can yield meaningful scientific insights remains debated. Here, we discuss the choice of a mapping model in the context of three overarching desiderata: predictive accuracy, interpretability, and biological plausibility. We show that, contrary to popular intuition, these desiderata do not map cleanly onto the linear/nonlinear divide; instead, each desideratum can refer to multiple research goals, each of which imposes its own constraints on the mapping model. Moreover, we argue that, instead of categorically treating the mapping models as linear or nonlinear, we should instead aim to estimate the complexity of these models. We show that, in many cases, complexity provides a more accurate reflection of restrictions imposed by various research goals. Finally, we outline several complexity metrics that can be used to effectively evaluate mapping models.

neuroscience↗

bioRxiv2021

Functional selectivity for naturalistic social interaction perception in the human superior temporal sulcus

Recognizing others social interactions is a crucial human ability. Using simple stimuli, previous studies have shown that social interactions are selectively processed in the superior temporal sulcus (STS), but prior work with movies has suggested that social interactions are processed in the medial prefrontal cortex (mPFC), part of the theory of mind network. It remains unknown to what extent social interaction selectivity is observed in real world stimuli when controlling for other covarying perceptual and social information, such as faces, voices, and theory of mind. The current study utilizes a functional magnetic resonance imaging (fMRI) movie paradigm and advanced machine learning methods to uncover the brain mechanisms uniquely underlying naturalistic social interaction perception. We analyzed two publicly available fMRI datasets, collected while both male and female human participants (n = 17 and 18) watched two different commercial movies in the MRI scanner. By performing voxel-wise encoding and variance partitioning analyses, we found that broad social-affective features predict neural responses in social brain regions, including the STS and mPFC. However, only the STS showed robust and unique selectivity specifically to social interactions, independent from other covarying features. This selectivity was observed across two separate fMRI datasets. These findings suggest that naturalistic social interaction perception recruits dedicated neural circuity in the STS, separate from the theory of mind network, and is a critical dimension of human social understanding. Significance StatementSocial interaction perception guides our daily behavior, yet it is unknown how our brain processes real-world social interaction scenes. Here, we demonstrate that social brain areas, including the superior temporal sulcus (STS) and medial prefrontal cortex (mPFC), are sensitive to broad social-affective information in naturalistic movies, replicating prior results with controlled paradigms. We show for the first time however, that the STS, but not mPFC, selectively processes social interactions in natural movies, independent of other co-occurring perceptual and social features, including motion, presence of faces, speech, and mentalizing about others. Our results suggest that social interaction perception is a crucial dimension of social understanding represented in the human brain.

neuroscience↗

bioRxiv2019

The speed of human social interaction perception

The ability to detect and understand other peoples social interactions is a fundamental part of the human visual experience that develops early in infancy and is shared with other primates. However, the neural computations underlying this ability remain largely unknown. Is the detection of social interactions a rapid perceptual process, or a slower post-perceptual inference? Here we used magnetoencephalography (MEG) decoding and computational modeling to ask whether social interactions can be detected via fast, feedforward processing. Subjects in the MEG viewed snapshots of visually matched real-world scenes containing a pair of people who were either engaged in a social interaction or acting independently. The presence versus absence of a social interaction could be read out from subjects MEG data spontaneously, even while subjects performed an orthogonal task. This readout generalized across different scenes, revealing abstract representations of social interactions in the human brain. These representations, however, did not come online until quite late, at 300 ms after image onset, well after the time period of feedforward visual processes. In a second experiment, we found that social interaction readout occurred at this same latency even when subjects performed an explicit task detecting social interactions. Consistent with these latency results, a standard feedforward deep neural network did not contain an abstract representation of social interactions at any model layer. We further showed that MEG responses distinguished between different types of social interactions (mutual gaze vs joint attention) even later, around 500 ms after image onset. Taken together, these results suggest that the human brain spontaneously extracts the presence and type of others social interactions, but does so slowly, likely relying on iterative top-down computations.

neuroscience↗

bioRxiv2018

How face perception unfolds over time

Within a fraction of a second of viewing a face, we have already determined its gender, age and identity. A full understanding of this remarkable feat will require a characterization of the computational steps it entails, along with the representations extracted at each. To this end, we used magnetoencephalography to measure the time course of neural responses to faces, thereby addressing two fundamental questions about how face processing unfolds over time. First, using representational similarity analysis, we found that facial gender and age information emerged before identity information, suggesting a coarse-to-fine processing of face dimensions. Second, identity and gender representations of familiar faces were enhanced very early on, indicating that the previously-reported behavioral benefit for familiar faces results from tuning of early feed-forward processing mechanisms. These findings start to reveal the time course of face perception in humans, and provide powerful new constraints on computational theories of face perception.

neuroscience↗