Search bioRxiv⌕ Search

Biology subjects

Konkle, T.

Publications and source records attributed to Konkle, T..

At least 19 recordsLinked to original sources
bioRxiv2026

Parametric neural control differentiates top neural network models of primate visual cortex

Leading deep neural network encoding models predict visual cortical responses with nearly indistinguishable accuracy, raising the strong inference that these models have converged on the same underlying brain-aligned parameterization of natural image space. Here we demonstrate that this is not the case. We introduce axis-aligned feature accentuation, which converts each model's fitted encoding axis into graded stimulus perturbations that are predicted to parametrically control neural firing within and beyond the natural-image range. We generated over 27,500 controller stimuli from ten leading vision models and presented them to five macaques in closed-loop experiments targeting early, mid-, and high-level visual areas. Despite matched natural image predictivity, models diverged strongly in their ability to control neural firing using accentuated stimuli, revealing that most model encoding axes failed to capture the precise tuning of their corresponding neurons. The two adversarially trained models showed a consistent advantage, though adversarial robustness was only weakly predictive of neural control across other models. Instead, control was better predicted by the spatial frequency structure of the input gradient: the distribution of pixels influencing each encoding axis. Overall, these results establish neural control via axis-aligned feature accentuation as a causal method to assess the alignment between how neurons and models parameterize the visual world.

neuroscience↗
bioRxiv2026

Edge-based natural image reconstruction provides a unified account of many lightness illusions

The human visual system transforms patterns of light into rich perceptual experiences, where what we see is a construction that goes beyond simple measurement. Lightness illusions--where identical parts of an image can appear dramatically different depending on context--provide a window into these processes. Here we leverage a deep learning framework to investigate the constructive processes that give rise to lightness illusions, introducing the core computational goal of edge-based image reconstruction. Specifically, we demonstrate that autoencoder models trained to reconstruct natural images based only on an edge-based image representation naturally recapitulate a wide range of lightness illusions, which were previously assumed to require distinct mechanisms, inference over lighting sources, and explicit three-dimensional scene representation. These results offer a simpler, unified account of diverse lightness phenomena as emerging naturally from surface filling-in mechanisms, and broadly provide a framework for understanding the computational principles that underlie our perception of the visual world. SIGNIFICANCE STATEMENTThe human visual system shows remarkably stable perception of objects under different viewing conditions, but it uses strategies that can be thwarted by clever visual illusions - for instance, the exact same object can appear as either white or black in different contexts. The most complex of these lightness illusions have long been taken as evidence that perception involves explicit inference about 3D scene geometry and lighting conditions. However, here we show that these illusions also emerge in deep learning models, trained simply to reconstruct natural images from sparse edge signals. Thus, our perception of the lightness of surfaces in our world may instead arise from a much more primitive computation -- reconstructing surface appearance from edge responses.

neuroscience↗
bioRxiv2026

Principles of coarse-scale functional organization in occipitotemporal cortex

Occipitotemporal cortex is known to process visually perceived objects, yet identifying general principles underlying its coarse-scale functional organization remains challenging for two reasons. First, previous work leaves open whether proposed organizational dimensions, such as animacy or real-world size, are useful for characterizing high-level vision and whether they generalize to more diverse, naturalistic stimuli. Second, object properties are often highly intercorrelated, making it difficult to determine which dimensions, if any, dominate coarse-scale occipitotemporal cortex organization. We addressed these challenges through detailed analyses of the cortical topography underlying 15 object properties, using densely-sampled fMRI responses to thousands of natural images, and focusing on animacy and size given their established prominence. Although our results confirmed many characteristics of the purported animacy-size organization, they revealed distinctions that challenge dominant notions of this organization, highlighting the importance of generalizing to large-scale naturalistic data. Beyond animacy and size, we found that many of the 15 properties could alternatively serve as organizational dimensions of occipitotemporal cortex, yet none emerged as clearly dominant across all OTC. Together, our findings reveal a more complex animacy-size organization than previously assumed and highlight challenges of assigning individual principles to account for OTC organization.

neuroscience↗
bioRxiv2025

Monkey See, Model Knew: Large Language Models Accurately Predict Visual Brain Responses in Humans and Non-Human Primates

AO_SCPLOWBSTRACTC_SCPLOWRecent progress in multimodal AI and language-aligned visual representation learning has rekindled debates about the role of language in shaping the human visual system. In particular, the emergent ability of language-aligned vision models (e.g. CLIP) - and even pure language models (e.g. BERT) - to predict image-evoked brain activity has led some to suggest that human visual cortex itself may be language-aligned in comparable ways. But what would we make of this claim if the same procedures could model visual activity in a species without language? Here, we conducted controlled comparisons of pure-vision, pure-language, and multimodal vision-language models in their prediction of human (N=4) and rhesus macaque (N=6, 5:IT, 1:V1) ventral visual activity to the same set of 1000 captioned natural images (the NSD1000). The results revealed markedly similar patterns in model predictivity of early and late ventral visual cortex across both species. This suggests that language model predictivity of the human visual system is not necessarily due to the evolution or learning of language perse, but rather to the statistical structure of the visual world that is reflected in natural language.

neuroscience↗
bioRxiv2024

A feedforward mechanism for human-like contour integration

Deep neural network models provide a powerful experimental platform for exploring core mechanisms underlying human visual perception, such as perceptual grouping and contour integration -- the process of linking local edge elements to arrive at a unified perceptual representation of a complete contour. Here, we demonstrate that feedforward, nonlinear convolutional neural networks (CNNs) can emulate this aspect of human vision without relying on mechanisms proposed in prior work, such as lateral connections, recurrence, or top-down feedback. We identify two key inductive biases that give rise to human-like contour integration in purely feedforward CNNs: a gradual progression of receptive field sizes with increasing layer depth, and a bias towards relatively straight (gradually curved) contours. While lateral connections, recurrence, and feedback are ubiquitous and important visual processing mechanisms, these results provide a computational existence proof that a feedforward hierarchy is sufficient to implement gestalt "good continuation" mechanisms that detect extended contours in a manner that is consistent with human perception.

animal behavior and cognition↗
bioRxiv2024

Self-organized emergence of modularity, hierarchy, and mirror reversals from competitive synaptic growth in a developmental model of the visual pathway

A hallmark of the primate visual system is its architectural organization consisting of multiple distinct (modular) areas that connect hierarchically. These areas exhibit specific spatial organization on the cortical sheet, with primary visual cortex at the center and subsequent regions in the hierarchy encircling the earlier one, and detailed topological organization, with retinotopy in each area but striking mirror reversals across area boundaries. The developmental rules that drive the simultaneous formation of these architectural, spatial, and topographic aspects of organization are unknown. Here we demonstrate that a simple synaptic growth rule driven by spontaneous activity and heterosynaptic competition generates a detailed connectome of the visual pathway, with emergence of all three types of organization. We identify a theoretical principle -- local greedy wiring minimization via spontaneous drive (GWM-S) -- implemented by the mechanism, and use this insight to propose biologically distinct growth rules that predict similar endpoints but testably distinguishable developmental trajectories. The same rules predict how input geometry and cortical geometry together drive emergence of hierarchical, convolution-like, spatially and topographically organized sensory processing pathways for different modalities and species, providing a possible explanation for the observed pluripotency of cortical structure formation. We find that the few parameters governing structure emergence in the growth rule constitute simple knobs for rich control, that could (potentially genetically) encode a projection neuron-like connectivity patterns and interneuron-like ones. In all, the presented rules provide a parsimonious mechanistic model for the organization of sensory cortical hierarchies even without detailed genetic cues for features like map reversal, and provide numerous predictions for experiment during normal and perturbed development.

neuroscience↗
bioRxiv2023

A contrastive coding account of category selectivity in the ventral visual stream

Modular and distributed coding theories of category selectivity along the human ventral visual stream have long existed in tension. Here, we present a reconciling framework - contrastive coding - based on a series of analyses relating category selectivity within biological and artificial neural networks. We discover that, in models trained with contrastive self-supervised objectives over a rich natural image diet, category-selective tuning naturally emerges for faces, bodies, scenes, and words. Further, lesions of these model units lead to selective, dissociable recognition deficits, highlighting their distinct functional roles in information processing. Finally, these pre-identified units can predict neural responses in all corresponding face-, scene-, body-, and word-selective regions of human visual cortex, under a highly constrained sparse-positive encoding procedure. The success of this single model indicates that brain-like functional specialization can emerge without category-specific learning pressures, as the system learns to untangle rich image content. Contrastive coding, therefore, provides a unifying account of object category emergence and representation in the human brain.

neuroscience↗
bioRxiv2023

Contributions of early and mid-level visual cortex to high-level object categorization

The complexity of visual features for which neurons are tuned increases from early to late stages of the ventral visual stream. Thus, the standard hypothesis is that high-level functions like object categorization are primarily mediated by higher visual areas because they require more complex image formats that are not evident in early visual processing stages. However, human observers can categorize images as objects or animals or as big or small even when the images preserve only some low- and mid-level features but are rendered unidentifiable ( texforms, Long et al., 2018). This observation suggests that even the early visual cortex, in which neurons respond to simple stimulus features, may already encode signals about these more abstract high-level categorical distinctions. We tested this hypothesis by recording from populations of neurons in early and mid-level visual cortical areas while rhesus monkeys viewed texforms and their unaltered source stimuli (simultaneous recordings from areas V1 and V4 in one animal and separate recordings from V1 and V4 in two others). Using recordings from a few dozen neurons, we could decode the real-world size and animacy of both unaltered images and texforms. Furthermore, this neural decoding accuracy across stimuli was related to the ability of human observers to categorize texforms by real-world size and animacy. Our results demonstrate that neuronal populations early in the visual hierarchy contain signals useful for higher-level object perception and suggest that the responses of early visual areas to simple stimulus features display preliminary untangling of higher-level distinctions.

neuroscience↗
bioRxiv2023

Ultra-wide angle neuroimaging: insights into immersive scene representation

While humans experience the visual environment in a panoramic 220{degrees} view, traditional functional MRI setups are limited to display images like postcards in the central 10-15{degrees} of the visual field. Thus, it remains unknown how a scene is represented in the brain when perceived across the full visual field. Here, we developed a novel method for ultra-wide angle visual presentation and probed for signatures of immersive scene representation. To accomplish this, we bounced the projected image off angled-mirrors directly onto a custom-built curved screen, creating an unobstructed view of 175{degrees}. Scene images were created from custom-built virtual environments with a compatible wide field-of-view to avoid perceptual distortion. We found that immersive scene representation drives medial cortex with far-peripheral preferences, but surprisingly had little effect on classic scene regions. That is, scene regions showed relatively minimal modulation over dramatic changes of visual size. Further, we found that scene and face-selective regions maintain their content preferences even under conditions of central scotoma, when only the extreme far-peripheral visual field is stimulated. These results highlight that not all far-peripheral information is automatically integrated into the computations of scene regions, and that there are routes to high-level visual areas that do not require direct stimulation of the central visual field. Broadly, this work provides new clarifying evidence on content vs. peripheral preferences in scene representation, and opens new neuroimaging research avenues to understand immersive visual representation.

neuroscience↗
bioRxiv2022

Visual object topographic motifs emerge from self-organization of a unified representational space

The object-responsive cortex of the visual system has a highly systematic topography, with a macro-scale organization related to animacy and the real-world size of objects, and embedded meso-scale regions with strong selectivity for a handful of object categories. Here, we use self-organizing principles to learn a topographic representation of the data manifold of a deep neural network representational space. We find that a smooth mapping of this representational space showed many brain-like motifs, with (i) large-scale organization of animate vs. inanimate and big vs. small response preferences, supported by (ii) feature tuning related to textural and coarse form information, with (iii) naturally emerging face- and scene-selective regions embedded in this larger-scale organization. While some theories of the object-selective cortex posit that these differently tuned regions of the brain reflect a collection of distinctly specified functional modules, the present work provides computational support for an alternate hypothesis that the tuning and topography of the object-selective cortex reflects a smooth mapping of a unified representational space.

neuroscience↗
bioRxiv2022

Large-Scale Benchmarking of Diverse Artificial Vision Models in Prediction of 7T Human Neuroimaging Data

The rapid development and open-source release of highly performant computer vision models offers new potential for examining how different inductive biases impact representation learning and emergent alignment with the high-level human ventral visual system. Here, we assess a diverse set of 224 models, curated to enable controlled comparison of different model properties, testing their brain predictivity using large-scale functional magnetic resonance imaging data. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and markedly different task objectives (e.g. purely visual contrastive learning versus vision-language alignment) achieve near equivalent degrees of brain predictivity, when other factors are held constant. Instead, variation across model visual training diets yields the largest, most consistent effect on emergent brain predictivity. Overarching model properties commonly suspected to increase brain predictivity (e.g. greater effective dimensionality; learnable parameter count) were not robust indicators across this more extensive survey. We highlight that standard model-to-brain linear re-weighting methods may be too flexible, as most performant models have very similar brain-predictivity scores, despite significant variation in their underlying representations. Broadly, our findings point to the importance of visual diet, challenge common assumptions about the methods used to link models to brains, and more concretely outline future directions for leveraging the full diversity of existing open-source models as tools to probe the common computational principles underlying biological and artificial visual systems.

neuroscience↗
bioRxiv2022

The neural code for 'face cells' is not face specific

1 Face cells are visual neurons that respond more to faces than other objects. Clustered together in inferotemporal cortex, they are thought to carry out face processing specifically and are thus studied using faces almost exclusively. Analyzing neural responses in and around macaque face patches to hundreds of objects, we found graded response profiles for non-faces that were predictive of the degree of face selectivity and provided information on face-cell tuning that could not be characterized with actual faces. This relationship between non-face and face responses was not predicted by color and simple shape properties, but by information encoded in deep neural networks trained on general object classification rather than face identification. These findings contradict the long-standing assumption that face cells owe their category selectivity to face-specific features, instead providing evidence for the notion that category-selective neurons are best understood as tuning directions in an integrated, domain-general object space.

neuroscience↗
bioRxiv2022

Mid-level feature differences underlie early animacy and object size distinctions: Evidence from EEG decoding

Responses to visually-presented objects along the cortical surface of the human brain have a large-scale organization reflecting the broad categorical divisions of animacy and object size. Mounting evidence indicates that this topographical organization is driven by differences between objects in mid-level perceptual features. With regard to the timing of neural responses, images of objects quickly evoke neural responses with decodable information about animacy and object size, but are mid-level features sufficient to evoke these rapid neural responses? Or is slower iterative neural processing required to untangle information about animacy and object size from mid-level features? To answer this question, we used electroencephalography (EEG) to measure human neural responses to images of objects and their texform counterparts - unrecognizable images which preserve some mid-level feature information about texture and coarse form. We found that texform images evoked neural responses with early decodable information about both animacy and real-world size, as early as responses evoked by original images. Further, successful cross-decoding indicates that both texform and original images evoke information about animacy and size through a common underlying neural basis. Broadly, these results indicate that the visual system contains a mid-level feature bank carrying linearly decodable information on animacy and size, which can be rapidly activated without requiring explicit recognition or protracted temporal processing.

neuroscience↗
bioRxiv2022

Ramp-shaped neural tuning supports graded population-level representation of the object-to-scene continuum

We can easily perceive the spatial scale depicted in a picture, regardless of whether it is a small space (e.g., a close-up view of a chair) or a much larger space (e.g., an entire class room). How does the human visual system encode this continuous dimension? Here, we investigated the underlying neural coding of depicted spatial scale, by examining the voxel tuning and topographic organization of brain responses. We created naturalistic yet carefully-controlled stimuli by constructing virtual indoor environments, and rendered a series of snapshots to smoothly sample between a close-up view of the central object and far-scale view of the full environment (object-to-scene continuum). Human brain responses were measured to each position using functional magnetic resonance imaging. We did not find evidence for a smooth topographic mapping for the object-to-scene continuum on the cortex. Instead, we observed large swaths of cortex with opposing ramp-shaped profiles, with highest responses to one end of the object-to-scene continuum or the other, and a small region showing a weak tuning to intermediate scale views. Importantly, when we considered the multi-voxel patterns of the entire ventral occipito-temporal cortex, we found smooth and linear representation of the object-to-scene continuum. Thus, our results together suggest that depicted spatial scale is coded parametrically in large-scale population codes across the entire ventral occipito-temporal cortex.

neuroscience↗
bioRxiv2021

How big should this object be? Perceptual influences on viewing-size preferences

When viewing objects depicted in a frame, observers prefer to view large objects like cars in larger sizes and smaller objects like cups in smaller sizes. That is, the visual size of an object that "looks best" is linked to its typical physical size in the world. Why is this the case? One intuitive possibility is that these preferences are driven by semantic knowledge: For example, when we recognize a sofa, we access our knowledge about its real-world size, and this influences what size we prefer to view the sofa within a frame. However, might visual processing play a role in this phenomenon--that is, do visual features that are related to big and small objects look better at big and small visual sizes, respectively, even when observers do not have explicit access to semantic knowledge about the objects? To test this possibility, we used "texform" images, which are synthesized versions of recognizable objects, which critically retain local perceptual texture and coarse contour information, but are no longer explicitly recognizable. To test for visual size preferences, we first used a size adjustment task, and the results were equivocal. However, clear results were obtained using a two-interval forced choice task, in which each texform was presented at the preferred visual size of its corresponding original image, and a visual size slightly bigger or smaller. Observers consistently selected the texform presented at the canonical visual size as the more aesthetically pleasing one. An additional control experiment ruled out alternative explanations related to size priming effects. These results suggest that the preferred visual size of an object depends not only on explicit knowledge of its real-world size, but also can be evoked by mid-level visual features that systematically covary with an objects real-world size. HighlightsO_LIWe prefer to view large objects like cars large, and small objects like cups small C_LIO_LIIntuitively, such preferences may be driven by our knowledge of object sizes C_LIO_LIWe used unrecognizable texforms of objects that preserved mid-level visual features C_LIO_LISimilar viewing size preferences can be revealed with these texforms C_LIO_LISuch preferences thus arise not only from knowledge but also from visual processing C_LI

animal behavior and cognition↗
bioRxiv2021

Beyond category-supervision: instance-level contrastive learning models predict human visual system responses to objects

Anterior regions of the ventral visual stream have substantial information about object categories, prompting theories that category-level forces are critical for shaping visual representation. The strong correspondence between category-supervised deep neural networks and ventral stream representation supports this view, but does not provide a viable learning model, as these deepnets rely upon millions of labeled examples. Here we present a fully self-supervised model which instead learns to represent individual images, where views of the same image are embedded nearby in a low-dimensional feature space, distinctly from other recently encountered views. We find category information implicitly emerges in the feature space, and critically that these models achieve parity with category-supervised models in predicting the hierarchical structure of brain responses across the human ventral visual stream. These results provide computational support for learning instance-level representation as a viable goal of the ventral stream, offering an alternative to the category-based framework that has been dominant in visual cognitive neuroscience.

neuroscience↗
bioRxiv2021

General object-based features account for letter perception better than specialized letter features

After years of experience, humans become experts at perceiving letters. Is this visual capacity attained by learning specialized letter features, or by reusing general visual features previously learned in service of object categorization? To explore this question, we first measured the perceptual similarity of letters in two behavioral tasks, visual search and letter categorization. Then, we trained deep convolutional neural networks on either 26-way letter categorization or 1000-way object categorization, as a way to operationalize possible specialized letter features and general object-based features, respectively. We found that the general object-based features more robustly correlated with the perceptual similarity of letters. We then operationalized additional forms of experience-dependent letter specialization by altering object-trained networks with varied forms of letter training; however, none of these forms of letter specialization improved the match to human behavior. Thus, our findings reveal that it is not necessary to appeal to specialized letter representations to account for perceptual similarity of letters. Instead, we argue that it is more likely that the perception of letters depends on domain-general visual features. AUTHOR SUMMARYFor over a century, scientists have conducted behavioral experiments to investigate how the visual system recognizes letters, but it has proven difficult to propose a model of the feature space underlying this capacity. Here we leveraged recent advances in machine learning to model a wide variety of features ranging from specialized letter features to general object-based features. Across two large-scale behavioral experiments we find that general object-based features account well for letter perception, and that adding letter specialization did not improve the correspondence to human behavior. It is plausible that the ability to recognize letters largely relies on general visual features unaltered by letter learning.

neuroscience↗
bioRxiv2021

Behavioral and Neural Representations en route to Intuitive Action Understanding

When we observe another persons actions, we process many kinds of information - from how their body moves to the intention behind their movements. What kinds of information underlie our intuitive understanding about how similar actions are to each other? To address this question, we measured the intuitive similarities among a large set of everyday action videos using multi-arrangement experiments, then used a modeling approach to predict this intuitive similarity space along three hypothesized properties. We found that similarity in the actors inferred goals predicted the intuitive similarity judgments the best, followed by similarity in the actors movements, with little contribution from the videos visual appearance. In opportunistic fMRI analyses assessing brain-behavior correlations, we found suggestive evidence for an action processing hierarchy, in which these three kinds of action similarities are reflected in the structure of brain responses along a posterior-to-anterior gradient on the lateral surface of the visual cortex. Altogether, this work joins existing literature suggesting that humans are naturally tuned to process others intentions, and that the visuo-motor cortex computes the perceptual precursors of the higher-level representations over which intuitive action perception operates.

neuroscience↗