Biology subjects

Kriegeskorte, N.

Publications and source records attributed to Kriegeskorte, N..

At least 19 recordsLinked to original sources

bioRxiv2026

Lateral recurrence dynamically refines visual representations for task performance

Recurrent connections are pervasive in visual cortex, and visual representations evolve as recurrent processing unfolds. Yet the mechanism underlying this temporal transformation remains poorly understood. To address this question, we trained recurrent convolutional networks on face and object recognition and tested them on a parametric stimulus set in which category, identity, and pose were systematically varied. This design isolates how lateral recurrence reshapes neural-population representational geometry and single-unit tuning across recurrent timesteps. Within the trained domain, recurrence increased the separability between identities in deeper layers, while leaving the non-trained domain representations largely unchanged. Gating lateral weights abolished the across-timestep gain in trained-domain identity decoding, demonstrating that lateral recurrence drives this refinement. At the single-unit level, recurrence strengthened the correlation between each unit's category preference and its within-domain identity sensitivity, providing a unit-level mechanism for the population-level changes. These results identify lateral recurrence as a dynamic mechanism for refining learned representational structure within a fixed neural population, yielding testable predictions for time-resolved recordings in domain-selective cortex.

neuroscience↗

bioRxiv2026

Foveation as self-supervision signal: Blur masks beat blank masks for learning robust visual representations

The primate retina is strikingly non-uniform, with receptor density falling off sharply from fovea to periphery. Every eye-movement brings a previously low-acuity peripheral area to high-acuity foveal processing. We hypothesize that this process provides a natural self-supervision signal, enabling representation learning by predicting fine details in the periphery. We implement this in a SimMIM-style vision transformer, replacing its blank masks (ViT-Blank) with a uniform blur mask (ViT-Blur) and further with retinal masks in which a foveal patch is randomly selected, and all other patches are progressively blurred based on eccentricity from the foveal patch (ViT-Retina). ViTs were pre-trained to reconstruct the full-resolution image from the corrupted input and then fine-tuned on clean images for classification. ViT-Blur and ViT-Retina outperform ViT-Blank on the pre-training reconstruction task while controlling for retained information from the input across corruption types. Critically, they also outperform ViT-Blank on downstream classification, showing that they learn representations that generalize better. The ViT-Blur and ViT-Retina are also more robust in impoverishment conditions such as higher mask ratios, fewer pre-training epochs, smaller training sets, and test time image corruptions. Using additive band-limited noise, we discovered that ViT-Blank relies on higher spatial frequencies compared to ViT-Blur and ViT-Retina, and we show this to be the likely cause of poorer generalization: the centroid of a model's spatial frequency tuning is negatively correlated with its classification accuracy ($r = -0.73$). We further show that a closed-form linear-ridge encoder derived from the same reconstruction objective reproduces this tuning ($r = 0.88$). Overall, our experiments show the promise of a biologically-inspired self-supervision objective for learning robust visual representations.

neuroscience↗

bioRxiv2026

How attention saves energy in vision

Attention has long been thought to enable efficient vision,1-8 yet it requires additional neural machinery and energy. Whether attention yields net energetic benefits--after accounting for the cost of control--has never been demonstrated. Here we show that attentional control can substantially improve whole-system energy efficiency in a model of primate visual processing. Our model, EAN ("Energy-efficient Attention Network"), implements attention as recurrent top-down multiplicative gain over features, space, and time. EAN is optimized using a joint objective combining task performance and neurobiologically grounded energy costs accounting for action potentials and synaptic transmission across all components,9-11 including the attentional control circuitry itself. On a visual-category-search task requiring joint identification and localization of a target, EAN learns to focus its energy dynamically on task-relevant locations and features, reducing total energy use by up to 50% at matched accuracy and enabling flexible trial-by-trial trading of accuracy against energy. The model variant combining feature-based and spatial attention is most efficient and best captures human errors and difficulty judgments. EAN generalizes to classical attention tasks, replicating canonical effects of attention on firing rates, variability, and noise correlations,12 and patterns of V4-to-V1 feedback suppression.13 Our work connects a cognitive function (attention), a neural mechanism (gain modulation), and a neurobiological constraint (metabolic cost) in a single mechanistic model that explains how selection and recurrence enable flexible, energy-efficient vision.

neuroscience↗

bioRxiv2026

Neural representations supporting generalization under continual learning

Abstraction and generalization are essential for flexible decision-making in novel situations. Recent work in humans and monkeys has shown how abstract variables are encoded by the representational geometry of neural population activity. However, these observations--which are typically made after learning has converged--demonstrate the product of abstraction, but not the process by which abstract knowledge is learned: how are the inputs from concrete experiences transformed into abstract knowledge, and how do neural circuits perform these operations and relay this knowledge? To address these questions, we developed a factorized model of temporal abstraction that builds on the successor representation. The model disentangles the contributions of different levels of abstract learning--from stimulus-stimulus associations to a generalizable task schema--in the form of a factorized prediction error that relates the change in relational knowledge to a predicted change in representational geometry on each trial. We fit the model to the behavior of human participants performing a context-dependent decision task during fMRI. The model captured the learning dynamics at multiple timescales, including the increasing contribution of generalization as participants transferred abstracted relational knowledge between novel task instances. In fMRI, BOLD activity in hippocampus--where, in past work, abstract knowledge was represented after learning--was increasingly attributed to the acquisition of abstract knowledge based on generalization. A similar temporal pattern was observed in entorhinal cortex, a putative source of low-dimensional structural information, and orbitofrontal cortex (OFC), which may depend on relational knowledge to represent state relationships as a cognitive map that guides choices. Indeed, individual variation in the generalization signal in OFC correlated with behavioral performance on key trials that required relational knowledge. Our findings show how the brain regions previously shown to represent abstract knowledge after learning also support the process of abstraction as it evolves from learning concrete associations to a generalizable schema. Our approach offers a computational framework for disentangling the operations driving abstract learning and probing their neural correlates in the dynamics of representational geometry.

neuroscience↗

bioRxiv2025

Framed RSA: Representational comparisons that honor bothgeometry and population-mean response preferences

Representational similarity analysis (RSA) characterizes the geometry of neural activity patterns elicited by different stimuli while discarding information about neural response preferences, regional population-mean activity, and the absolute location and orientation of the patterns in the multivariate response space. When evaluating alternative representational models, invariance to certain aspects of the neural code is desirable because systems might use superficially different encodings to implement the same computations. However, neural preferences and regional-mean activation are arguably physiologically and mechanistically important, and so we may want our models to predict them correctly. Here we introduce a novel analysis technique, framed RSA, which honors both geometry and population-mean preferences in evaluating model-predicted representations. To achieve this, we augment the set of patterns that define the geometry by two reference patterns: the all-zero point (origin) and an all-c (uniform constant) pattern in the multivariate response space, enabling RSA to incorporate information about the profile across stimuli of regional population-mean activations and about the global location and orientation of the ensemble of response patterns. We show that framed RSA improves model-selection accuracy when ground truth is known, considering brain-region identification (using human fMRI data from the Natural Scenes Dataset and macaque intracranial recording data from the Things Ventral Stream Spiking Dataset) and deep-neural-network-layer identification. By incorporating neural population preferences into model evaluation, framed RSA enables more mechanistically meaningful model comparisons and benefits from improved power for model-comparative inference.

neuroscience↗

bioRxiv2025

A Python Toolbox for Representational Similarity Analysis

Representational similarity analysis (RSA) is a method to characterize neural representations and evaluate computational models based on neural representational geometries. Here we present a wave of recent methodological advances, including improved measures of representational distances, evaluators for representational models, and statistical inference methods, which are available to the community in a new open-source toolbox in Python. The rsatoolbox enables neuroscientists to explore neural representational geometries and to evaluate neural network models, connecting theory to experiment in the new era of big models and big data.

neuroscience↗

bioRxiv2024

When do measured representational distances reflect the neural representational geometry?

The representational geometry of a brain region can be characterized by the distances among neural activity patterns for a set of experimental conditions. Researchers routinely estimate representational distances from brain-activity measurements that either sparsely sample the underlying neural population (e.g. neural recordings) or pool across the activity of many neurons (e.g. fMRI voxels). Here we use theory and simulations to clarify under what circumstances representational distances estimated from brain-activity measurements reflect the representational geometry of the underlying neural population, and what distortions must be expected under other circumstances. We demonstrate that the estimated representational distances are undistorted if single neurons are sampled at random. For voxels that take non-negatively weighted linear combinations, the resulting geometry is linearly distorted, correctly reflecting the population-mean dimension, while downscaling all orthogonal dimensions, for which the averaging cancels a large portion of the signal. Surprisingly, removing the mean from voxel patterns recovers the underlying representational geometry exactly in expectation under idealized conditions. This explains why the correlation distance, the most popular measure of representational dissimilarity in neuroimaging studies, "works" so well, yielding geometries that can appear similar between fMRI and neural recordings. The Euclidean (or Mahalanobis) distance computed after removing the mean of each pattern (without normalizing its variance) is an attractive alternative to the correlation distance in that it corrects for the inflated relative contribution of the population-mean dimension, while avoiding the drawback of the correlation distance: it can be large for confusable low-norm patterns, failing to reflect decodability. Our results demonstrate that measured representational distances reflect the neural representational geometry when (1) single neurons are sampled at random or (2) the weights with which the measured responses sample the neurons are drawn i.i.d. and (2a) the weights are drawn from a zero-mean distribution or (2b) the population mean is the same for all conditions or (2c) the mean is removed from each estimated pattern. We discuss practical implications for analyses of neural representational geometries.

neuroscience↗

bioRxiv2023

Iterative convergent computation may not be a useful inductive bias for residual neural networks

Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. Here we quantify the degree to which ResNets learn iterative solutions and introduce a regularization approach that encourages learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a "recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. However, neither recurrence regularization nor spectral normalization improve classification accuracy on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or on challenging recognition tasks with partial occlusions (Digitclutter). Iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets.

neuroscience↗

bioRxiv2023

Predicting brain activity using Transformers

The Algonauts challenge [Gifford et al., 2023] called on the community to provide novel solutions for predicting brain activity of humans viewing natural scenes. This report provides an overview and technical details of our submitted solution. We use a general transformer encoder-decoder model to map images to fMRI responses. The encoder model is a vision transformer trained using self-supervised methods (DINOv2). The decoder uses queries corresponding to different brain regions of interests (ROI) in different hemispheres to gather relevant information from the encoder output for predicting neural activity in each ROI. The output tokens from the decoder are then linearly mapped to the fMRI activity. The predictive success (challenge score: 63.5229, rank 2) suggests that features from self-supervised transformers may deserve consideration as models of human visual brain representations and shows the effectiveness of transformer mechanisms (self and cross-attention) to learn the mapping from features to brain responses. Code is available in this github repository.

neuroscience↗

bioRxiv2023

TorchLens: A Python package for extracting and visualizing hidden activations of PyTorch models

Deep neural network models (DNNs) are essential to modern AI and provide powerful models of information processing in biological neural networks. Researchers in both neuroscience and engineering are pursuing a better understanding of the internal representations and operations that undergird the successes and failures of DNNs. Neuroscientists additionally evaluate DNNs as models of brain computation by comparing their internal representations to those found in brains. It is therefore essential to have a method to easily and exhaustively extract and characterize the results of the internal operations of any DNN. Many models are implemented in PyTorch, the leading framework for building DNN models. Here we introduce TorchLens, a new open-source Python package for extracting and characterizing hidden-layer activations in PyTorch models. Uniquely among existing approaches to this problem, TorchLens has the following features: (1) it exhaustively extracts the results of all intermediate operations, not just those associated with PyTorch module objects, yielding a full record of every step in the models computational graph, (2) it provides an intuitive visualization of the models complete computational graph along with metadata about each computational step in a models forward pass for further analysis, (3) it contains a built-in validation procedure to algorithmically verify the accuracy of all saved hidden-layer activations, and (4) the approach it uses can be automatically applied to any PyTorch model with no modifications, including models with conditional (if-then) logic in their forward pass, recurrent models, branching models where layer outputs are fed into multiple subsequent layers in parallel, and models with internally generated tensors (e.g., injections of noise). Furthermore, using TorchLens requires minimal additional code, making it easy to incorporate into existing pipelines for model development and analysis, and useful as a pedagogical aid when teaching deep learning concepts. We hope this contribution will help researchers in AI and neuroscience understand the internal representations of DNNs.

neuroscience↗

bioRxiv2023

Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks

Primates can recognize objects despite 3D geometric variations such as in-depth rotations. The computational mechanisms that give rise to such invariances are yet to be fully understood. A curious case of partial invariance occurs in the macaque face-patch AL and in fully connected layers of deep convolutional networks in which neurons respond similarly to mirror-symmetric views (e.g., left and right profiles). Why does this tuning develop? Here, we propose a simple learning-driven explanation for mirror-symmetric viewpoint tuning. We show that mirror-symmetric viewpoint tuning for faces emerges in the fully connected layers of convolutional deep neural networks trained on object recognition tasks, even when the training dataset does not include faces. First, using 3D objects rendered from multiple views as test stimuli, we demonstrate that mirror-symmetric viewpoint tuning in convolutional neural network models is not unique to faces: it emerges for multiple object categories with bilateral symmetry. Second, we show why this invariance emerges in the models. Learning to discriminate among bilaterally symmetric object categories induces reflection-equivariant intermediate representations. AL-like mirror-symmetric tuning is achieved when such equivariant responses are spatially pooled by downstream units with sufficiently large receptive fields. These results explain how mirror-symmetric viewpoint tuning can emerge in neural networks, providing a theory of how they might emerge in the primate brain. Our theory predicts that mirror-symmetric viewpoint tuning can emerge as a consequence of exposure to bilaterally symmetric objects beyond the category of faces, and that it can generalize beyond previously experienced object categories.

neuroscience↗

bioRxiv2021

Deep neural networks and visuo-semantic models explain complementary components of human ventral-stream representational dynamics

Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural timeseries data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography (MEG) data acquired in human participants (9 females, 6 males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object rep-resentations unfold over space and time. While lower-level visual areas are better explained by DNN features, especially during the early phase of the response (< 128 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features during a later time window (starting 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured, and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral stream computations. SIGNIFICANCE STATEMENTWhen we view objects such as faces and cars in our visual environment, their neural representations dynamically unfold over time at a millisecond scale. These dynamics reflect the cortical computations that support fast and robust object recognition. Deep neural networks (DNNs) have emerged as a promising framework for modeling these computations but cannot yet fully account for the neural dynamics. Using magnetoencephalography data acquired in human observers during object viewing, we show that readily nameable aspects of objects, such as "eye", "wheel", and "face", can account for variance in the neural dynamics over and above DNNs. These findings suggest that DNNs and humans may in part rely on different object features for visual recognition and provide guidelines for model improvement.

neuroscience↗

bioRxiv2021

Disentangling dimensions of animacy in human brain and behaviour

Distinguishing animate from inanimate things is of great behavioural importance. Despite distinct brain and behavioural responses to animate and inanimate things, it remains unclear which object properties drive these responses. Here, we investigate the importance of five object dimensions related to animacy ("being alive", "looking like an animal", "having agency", "having mobility", and "being unpredictable") in brain (fMRI, EEG) and behaviour (property and similarity judgements) of 19 participants. We used a stimulus set of 128 images, optimized by a genetic algorithm to disentangle these five dimensions. The five dimensions explained much variance in the similarity judgments. Each dimension also explained a modest but significant amount of variance in the brain representations, except, surprisingly, "being alive". Different brain regions sensitive to animacy may represent distinct dimensions, either as accessible perceptual stepping stones toward detecting whether something is alive or because they are of behavioural importance in their own right.

neuroscience↗

bioRxiv2021

Category-specific representational patterns in left inferior frontal and temporal cortex reflect similarities and differences in the sensorimotor and distributional properties of concepts.

Neuronal populations code similar concepts by similar activity patterns across the human brains networks supporting language comprehension. However, it is unclear to what extent such meaning-to-symbol mapping reflects statistical distributions of symbol meanings in language use, as quantified by word co-occurrence frequencies, or, rather, experiential information thought to be necessary for grounding symbols in sensorimotor knowledge. Here we asked whether integrating distributional semantics with human judgments of grounded sensorimotor semantics better approximates the representational similarity of conceptual categories in the brain, as compared with each of these methods used separately. We examined the similarity structure of activation patterns elicited by action- and object-related concepts using multivariate representational similarity analysis (RSA) of fMRI data. The results suggested that a semantic vector integrating both sensorimotor and distributional information yields best category discrimination on the cognitive-linguistic level, and explains the corresponding activation patterns in left posterior inferior temporal cortex. In turn, semantic vectors based on detailed visual and motor information uncovered category-specific similarity patterns in fusiform and angular gyrus for object-related concepts, and in motor cortex, left inferior frontal cortex (BA 44), and supramarginal gyrus for action-related concepts.

neuroscience↗

bioRxiv2021

Face dissimilarity judgements are predicted by representational distance in deep neural networks and principal-component face space

Human vision is attuned to the subtle differences between individual faces. Yet we lack a quantitative way of predicting how similar two face images look, or whether they appear to show the same person. Principal-components-based 3D morphable models are widely used to generate stimuli in face perception research. These models capture the distribution of real human faces in terms of dimensions of physical shape and texture. How well does a "face space" defined to model the distribution of faces as an isotropic Gaussian explain human face perception? We designed a behavioural task to collect dissimilarity and same/different identity judgements for 232 pairs of realistic faces. The stimuli densely sampled geometric relationships in a face space derived from principal components of 3D shape and texture (Basel Face Model, BFM). We then compared a wide range of models in their ability to predict the data, including the BFM from which faces were generated, a 2D morphable model derived from face photographs, and image-computable models of visual perception. Euclidean distance in the BFM explained both similarity and identity judgements surprisingly well. In a comparison against 14 alternative models, we found that BFM distance was competitive with representational distances in state-of-the-art image-computable deep neural networks (DNNs), including a novel DNN trained on BFM identities. Models describing the distribution of facial features across individuals are not only useful tools for stimulus generation. They also capture important information about how faces are perceived, suggesting that human face representations are tuned to the statistical distribution of faces.

neuroscience↗

bioRxiv2020

FFA and OFA encode distinct types of face identity information

Faces of different people elicit distinct functional MRI (fMRI) patterns in several face-selective brain regions. Here we used representational similarity analysis to investigate what type of identity-distinguishing information is encoded in three face-selective regions: fusiform face area (FFA), occipital face area (OFA), and posterior superior temporal sulcus (pSTS). We used fMRI to measure brain activity patterns elicited by naturalistic videos of famous face identities, and compared their representational distances in each region with models of the differences between identities. Models included low-level to high-level image-computable properties and complex human-rated properties. We found that the FFA representation reflected perceived face similarity, social traits, and gender, and was well accounted for by the OpenFace model (deep neural network, trained to cluster faces by identity). The OFA encoded low-level image-based properties (pixel-wise and Gabor-jet dissimilarities). Our results suggest that, although FFA and OFA can both discriminate between identities, the FFA representation is further removed from the image, encoding higher-level perceptual and social face information.

neuroscience↗

bioRxiv2020

Diverse deep neural networks all predict human IT well, after training and fitting

Deep neural networks (DNNs) trained on object recognition provide the best current models of high-level visual areas in the brain. What remains unclear is how strongly network design choices, such as architecture, task training, and subsequent fitting to brain data contribute to the observed similarities. Here we compare a diverse set of nine DNN architectures on their ability to explain the representational geometry of 62 isolated object images in human inferior temporal (hIT) cortex, as measured with functional magnetic resonance imaging. We compare untrained networks to their task-trained counterparts, and assess the effect of fitting them to hIT using a cross-validation procedure. To best explain hIT, we fit a weighted combination of the principal components of the features within each layer, and subsequently a weighted combination of layers. We test all models across all stages of training and fitting for their correlation with the hIT representational dissimilarity matrix (RDM) using an independent set of images and subjects. We find that trained models significantly outperform untrained models (accounting for 57% more of the explainable variance), suggesting that features representing natural images are important for explaining hIT. Model fitting further improves the alignment of DNN and hIT representations (by 124%), suggesting that the relative prevalence of different features in hIT does not readily emerge from the particular ImageNet object-recognition task used to train the networks. Finally, all DNN architectures tested achieved equivalent high performance once trained and fitted. Similar ability to explain hIT representations appears to be shared among deep feedforward hierarchies of nonlinear features with spatially restricted receptive fields.

neuroscience↗

bioRxiv2020

Noise ceiling on the crossvalidated performance of reweighted models of representational dissimilarity: Addendum to Khaligh-Razavi & Kriegeskorte (2014)

An error was made in including noise ceilings for human data in Khaligh-Razavi and Kriegeskorte (2014). For comparability with the macaque data, human data were averaged across participants before analysis. Therefore the noise ceilings indicating variability across human participants do not accurately depict the upper bounds of possible model performance and should not have been shown. Creating noise ceilings appropriate for the fitted models is not trivial. Below we present a method for doing this, and the results obtained with this new method. The corrected results differ from the original results in that the best-performing model (weighted combination of AlexNet layers and category readouts) does not reach the lower bound of the noise ceiling. However, the best-performing model is not significantly below the lower bound of the noise ceiling. The claim that the model "fully explains" the human IT data appears overstated. All other claims of the paper are unaffected.

neuroscience↗