Biology subjects

Sinz, F. H.

Publications and source records attributed to Sinz, F. H..

At least 19 recordsLinked to original sources

bioRxiv2025

Dual-feature selectivity enables bidirectional coding in visual cortical neurons

Sensory neurons are traditionally viewed as feature detectors that respond with an increase in firing rate to preferred stimuli while remaining unresponsive to others. Here, we identify a dual-feature encoding strategy in macaque visual cortex, wherein many neurons in areas V1 and V4 are selectively tuned to two distinct visual features--one that enhances and one that suppresses activity--around an elevated baseline firing rate. By combining neuronal recordings with functional digital twin models--deep learning-based predictive models of biological neurons--we were able to systematically identify each neurons preferred and non-preferred features. These feature pairs served as anchors for a continuous, low-dimensional axis in natural image similarity space, along which neuronal activity varied approximately linearly. Within a single visual area, visual features that strongly or weakly activated individual neurons also had a high probability of modulating the activity of other neurons, suggesting a shared feature selectivity across the population that structures stimulus encoding. We show that this encoding strategy is conserved across species, present in both primary and lateral visual areas of mouse cortex. Dual-feature selectivity is consistent with recent anatomical evidence for feature-specific inhibitory connectivity, complementing the feature-detector principle through circuit mechanisms in which selective excitation and inhibition may together enhance the representational capacity of the neuronal population.

neuroscience↗

bioRxiv2023

Energy Guided Diffusion for Generating Neurally Exciting Images

In recent years, most exciting inputs (MEIs) synthesized from encoding models of neuronal activity have become an established method to study tuning properties of biological and artificial visual systems. However, as we move up the visual hierarchy, the complexity of neuronal computations increases. Consequently, it becomes more challenging to model neuronal activity, requiring more complex models. In this study, we introduce a new attention readout for a convolutional data-driven core for neurons in macaque V4 that outperforms the state-of-the-art task-driven ResNet model in predicting neuronal responses. However, as the predictive network becomes deeper and more complex, synthesizing MEIs via straightforward gradient ascent (GA) can struggle to produce qualitatively good results and overfit to idiosyncrasies of a more complex model, potentially decreasing the MEIs model-to-brain transferability. To solve this problem, we propose a diffusion-based method for generating MEIs via Energy Guidance (EGG). We show that for models of macaque V4, EGG generates single neuron MEIs that generalize better across architectures than the state-of-the-art GA while preserving the within-architectures activation and requiring 4.7x less compute time. Furthermore, EGG diffusion can be used to generate other neurally exciting images, like most exciting natural images that are on par with a selection of highly activating natural images, or image reconstructions that generalize better across architectures. Finally, EGG is simple to implement, requires no retraining of the diffusion model, and can easily be generalized to provide other characterizations of the visual system, such as invariances. Thus EGG provides a general and flexible framework to study coding properties of the visual system in the context of natural images.1

neuroscience↗

bioRxiv2023

Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topological organization

Deciphering the brains structure-function relationship is key to understanding the neuronal mechanisms underlying perception and cognition. The cortical column, a vertical organization of neurons with similar functions, is a classic example of primate neocortex structure-function organization. While columns have been identified in primary sensory areas using parametric stimuli, their prevalence across higher-level cortex is debated, particularly regarding complex tuning in natural image space. However, a key hurdle in identifying columns is characterizing the complex, nonlinear tuning of neurons to high-dimensional sensory inputs. Building on prior findings of topological organization for features like color and orientation, we investigate functional clustering in macaque visual area V4 in non-parametric natural image space, using large-scale recordings and deep learning-based analysis. We combined linear probe recordings with deep learning methods to systematically characterize the tuning of >1,200 V4 neurons using in silico synthesis of most exciting images (MEIs), followed by in vivo verification. Single V4 neurons exhibited MEIs containing complex features, including textures and shapes, and even high-level attributes with eye-like appearance. Neurons recorded on the same silicon probe, inserted orthogonal to the cortical surface, often exhibited similarities in their spatial feature selectivity, suggesting a degree of functional organization along the cortical depth. We quantified MEI similarity using human psychophysics and distances in a contrastive learning-derived embedding space. Moreover, the selectivity of the V4 neuronal population showed evidence of clustering into functional groups of shared feature selectivity. These functional groups showed parallels with the feature maps of units in artificial vision systems, suggesting potential shared encoding strategies. These results demonstrate the feasibility and scalability of deep learning-based functional characterization of neuronal selectivity in naturalistic visual contexts, offering a framework for quantitatively mapping cortical organization across multiple levels of the visual hierarchy.

neuroscience↗

bioRxiv2023

Towards a Foundation Model of the Mouse Visual Cortex

The complexity of neural circuits makes it challenging to decipher the brains algorithms of intelligence. Recent break-throughs in deep learning have produced models that accurately simulate brain activity, enhancing our understanding of the brains computational objectives and neural coding. However, these models struggle to generalize beyond their training distribution, limiting their utility. The emergence of foundation models, trained on vast datasets, has introduced a new AI paradigm with remarkable generalization capabilities. We collected large amounts of neural activity from visual cortices of multiple mice and trained a foundation model to accurately predict neuronal responses to arbitrary natural videos. This model generalized to new mice with minimal training and successfully predicted responses across various new stimulus domains, such as coherent motion and noise patterns. It could also be adapted to new tasks beyond neural prediction, accurately predicting anatomical cell types, dendritic features, and neuronal connectivity within the MICrONS functional connectomics dataset. Our work is a crucial step toward building foundation brain models. As neuroscience accumulates larger, multi-modal datasets, foundation models will uncover statistical regularities, enabling rapid adaptation to new tasks and accelerating research.

neuroscience↗

bioRxiv2023

Bipartite invariance in mouse primary visual cortex

A primary goal of sensory systems is to extract robust and meaningful features that are invariant to variations in the sensory input. Characterizing these invariances at the neuronal level is crucial for understanding how the visual system supports generalization, but the high-dimensional nature of ecological stimuli poses major challenges. Consequently, our understanding of how the brain represents invariances has historically depended on a few examples, such as phase invariance to grating stimuli in V1 complex cells. Here, we leverage the inception loop paradigm --iterating between large-scale recordings, deep learning neuronal predictive models, and in silico experiments with in vivo verification--to characterize neuronal invariances in mouse V1. Using a neuronal predictive model, we synthesized Diverse Exciting Inputs (DEIs) that strongly drive target neurons while differing substantially in image space. These DEIs revealed a novel bipartite invariance: one portion of the receptive field encodes shift-invariant, high-frequency textures, while the other encodes a fixed, low-frequency spatial pattern. This subfield division aligned with object boundaries defined by spatial frequency differences in highly activating stimuli, suggesting bi-partite invariance contributes to segmentation. Our analysis of computational models and anatomical data from the MICrONS dataset revealed a hierarchical organization of excitatory neurons in mouse V1 Layers 2/3: We found that postsynaptic neurons exhibited greater invariance than their presynaptic inputs, while neurons with lower invariance formed more connections. These findings suggest a synaptic-level hierarchy that progressively increases neural invariance within the primary visual cortex. Intriguingly, similar high-low frequency bipartite patterns strongly activate certain units in artificial neural networks, suggesting that universal visual representations govern both biological and artificial systems, potentially aiding in the extraction of visual features from complex backgrounds.

neuroscience↗

bioRxiv2023

Pattern completion and disruption characterize contextual modulation in mouse visual cortex

Vision is fundamentally context-dependent, with neuronal responses influenced not just by local features but also by surrounding contextual information. In the visual cortex, studies using simple grating stimuli indicate that congruent stimuli--where the center and surround share the same orientation--are more inhibitory than when orientations are orthogonal, potentially serving redundancy reduction and predictive coding. Understanding these center-surround interactions in relation to natural image statistics is challenging due to the high dimensionality of the stimulus space, yet crucial for deciphering the neuronal code of real-world sensory processing. Utilizing large-scale recordings from mouse V1, we trained convolutional neural networks (CNNs) to predict and synthesize surround patterns that either optimally suppressed or enhanced responses to center stimuli, confirmed by in v ivo experiments. Contrary to the notion that congruent stimuli are suppressive, we found that surrounds that completed patterns based on natural image statistics were facilitatory, while disruptive surrounds were suppressive. Applying our CNN image synthesis method in macaque V1, we discovered that pattern completion within the near surround occurred more frequently with excitatory than with inhibitory surrounds, suggesting that our results in mice are conserved in macaques. Further, experiments and model analyses confirmed previous studies reporting the opposite effect with grating stimuli in both species. Using the MICrONS functional connectomics dataset, we observed that neurons with similar feature selectivity formed excitatory connections regardless of their receptive field overlap, aligning with the pattern completion phenomenon observed for excitatory surrounds. Finally, our empirical results emerged in a normative model of perception implementing Bayesian inference, where neuronal responses are modulated by prior knowledge of natural scene statistics. In summary, our findings identify a novel relationship between contextual information and natural scene statistics and provide evidence for a role of contextual modulation in hierarchical inference.

neuroscience↗

bioRxiv2022

Large-scale unsupervised discovery of excitatory morphological cell types in mouse visual cortex

Neurons in the neocortex exhibit astonishing morphological diversity which is critical for properly wiring neural circuits and giving neurons their functional properties. However, the organizational principles underlying this morphological diversity remain an open question. Here, we took a data-driven approach using graph-based machine learning methods to obtain a low-dimensional morphological "bar code" describing more than 30,000 excitatory neurons in mouse visual areas V1, AL and RL that were reconstructed from the millimeter scale MICrONS serial-section electron microscopy volume. Contrary to previous classifications into discrete morphological types (m-types), our data-driven approach suggests that the morphological landscape of cortical excitatory neurons is better described as a continuum, with a few notable exceptions in layers 5 and 6. Dendritic morphologies in layers 2-3 exhibited a trend towards a decreasing width of the dendritic arbor and a smaller tuft with increasing cortical depth. Inter-area differences were most evident in layer 4, where V1 contained more atufted neurons than higher visual areas. Moreover, we discovered neurons in V1 on the border to layer 5 which avoided deeper layers with their dendrites. In summary, we suggest that excitatory neurons morphological diversity is better understood by considering axes of variation than using distinct m-types.

neuroscience↗

bioRxiv2022

It takes neurons to understand neurons: Digital twins of visual cortex synthesize neural metamers

Metamers, images that are perceived as equal, are a useful tool to study representations of natural images in biological and artificial vision systems. We synthesized metamers for the mouse visual system by inverting a deep encoding model to find an image that matched the observed neural activity to the original presented image. When testing the resulting images in physiological experiments we found that they most closely reproduced the neural activity of the original image when compared to other decoding methods, even when tested in a different animal whose neural activity was not used to produce the metamer. This demonstrates that deep encoding models do capture general characteristic properties of biological visual systems and can be used to define a meaningful perceptual loss for the visual system.

neuroscience↗

bioRxiv2022

Quantitative Census of Local Somatic Features in Mouse Visual Cortex

Mammalian neocortex contains a highly diverse set of cell types. These types have been mapped systematically using a variety of molecular, electrophysiological and morphological approaches. Each modality offers new perspectives on the variation of biological processes underlying cell type specialization. Cellular scale electron microscopy (EM) provides dense ultrastructural examination and an unbiased perspective into the subcellular organization of brain cells, including their synaptic connectivity and nanometer scale morphology. It also presents a clear challenge for analysis to identify cell-types in data that contains tens of thousands of neurons, most of which have incomplete reconstructions. To address this challenge, we present the first systematic survey of the somatic region of all cells within a cubic millimeter of cortex using quantitative features obtained from EM. This analysis demonstrates a surprising sufficiency of the perisomatic region to identify cell-types, including types defined primarily based on their connectivity patterns. We then describe how this classification facilitates cell type specific connectivity characterization and locating cells with rare connectivity patterns in the dataset.

neuroscience↗

bioRxiv2022

Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks

Responses to natural stimuli in area V4 - a mid-level area of the visual ventral stream - are well predicted by features from convolutional neural networks (CNNs) trained on image classification. This result has been taken as evidence for the functional role of V4 in object classification. However, we currently do not know if and to what extent V4 plays a role in solving other computational objectives. Here, we investigated normative accounts of V4 (and V1 for comparison) by predicting macaque single-neuron responses to natural images from the representations extracted by 23 CNNs trained on different computer vision tasks including semantic, geometric, 2D, and 3D types of tasks. We found that V4 was best predicted by semantic classification features and exhibited high task selectivity, while the choice of task was less consequential to V1 performance. Consistent with traditional characterizations of V4 function that show its high-dimensional tuning to various 2D and 3D stimulus directions, we found that diverse non-semantic tasks explained aspects of V4 function beyond those captured by individual semantic tasks. Nevertheless, jointly considering the features of a pair of semantic classification tasks was sufficient to yield one of our top V4 models, solidifying V4s main functional role in semantic processing and suggesting that V4s affinity to 2D or 3D stimulus properties found by electrophysiologists can result from semantic functional goals.

neuroscience↗

bioRxiv2021

A flow-based latent state generative model of neural population responses to natural images

We present a joint deep neural system identification model for two major sources of neural variability: stimulus-driven and stimulus-conditioned fluctuations. To this end, we combine (1) state-of-the-art deep networks for stimulus-driven activity and (2) a flexible, normalizing flow-based generative model to capture the stimulus-conditioned variability including noise correlations. This allows us to train the model end-to-end without the need for sophisticated probabilistic approximations associated with many latent state models for stimulus-conditioned fluctuations. We train the model on the responses of thousands of neurons from multiple areas of the mouse visual cortex to natural images. We show that our model outperforms previous state-of-the-art models in predicting the distribution of neural population responses to novel stimuli, including shared stimulus-conditioned variability. Furthermore, it successfully learns known latent factors of the population responses that are related to behavioral variables such as pupil dilation, and other factors that vary systematically with brain area or retinotopic location. Overall, our model accurately accounts for two critical sources of neural variability while avoiding several complexities associated with many existing latent state models. It thus provides a useful tool for uncovering the interplay between different factors that contribute to variability in neural activity.

neuroscience↗

bioRxiv2021

Petascale neural circuit reconstruction: automated methods

3D electron microscopy (EM) has been successful at mapping invertebrate nervous systems, but the approach has been limited to small chunks of mammalian brains. To scale up to larger volumes, we have built a computational pipeline for processing petascale image datasets acquired by serial section EM, a popular form of 3D EM. The pipeline employs convolutional nets to compute the nonsmooth transformations required to align images of serial sections containing numerous cracks and folds, detect neuronal boundaries, label voxels as axon, dendrite, soma, and other semantic categories, and detect synapses and assign them to presynaptic and postsynaptic segments. The output of neuronal boundary detection is segmented by mean affinity agglomeration with semantic and size constraints. Pipeline operations are implemented by leveraging distributed and cloud computing. Intermediate results of the pipeline are held in cloud storage, and can be effortlessly viewed as images, which aids debugging. We applied the pipeline to create an automated reconstruction of an EM image volume spanning four visual cortical areas of a mouse brain. Code for the pipeline is publicly available, as is the reconstructed volume.

neuroscience↗

bioRxiv2021

Functional connectomics spanning multiple areas of mouse visual cortex

To understand the brain we must relate neurons functional responses to the circuit architecture that shapes them. Here, we present a large functional connectomics dataset with dense calcium imaging of a millimeter scale volume. We recorded activity from approximately 75,000 neurons in primary visual cortex (VISp) and three higher visual areas (VISrl, VISal and VISlm) in an awake mouse viewing natural movies and synthetic stimuli. The functional data were co-registered with a volumetric electron microscopy (EM) reconstruction containing more than 200,000 cells and 0.5 billion synapses. Subsequent proofreading of a subset of neurons in this volume yielded reconstructions that include complete dendritic trees as well the local and inter-areal axonal projections that map up to thousands of cell-to-cell connections per neuron. Here, we release this dataset as an open-access resource to the scientific community including a set of tools that facilitate data retrieval and downstream analysis. In accompanying papers we describe our findings using the dataset to provide a comprehensive structural characterization of cortical cell types1-3 and the most detailed synaptic level connectivity diagram of a cortical column to date2, uncovering unique cell-type specific inhibitory motifs that can be linked to gene expression data4. Functionally, we identify new computational principles of how information is integrated across visual space5, characterize novel types of neuronal invariances6 and bring structure and function together to decipher a general principle that wires excitatory neurons within and across areas7, 8.

neuroscience↗

bioRxiv2020

Distinct organization of two cortico-cortical feedback pathways

Neocortical feedback is critical for processes like attention, prediction, and learning. A mechanistic understanding of its function requires deciphering its cell-type wiring logic. Recent studies revealed a disinhibitory circuit between motor and sensory areas in mice, where feedback preferentially targets vasointestinal peptide-expressing interneurons, in addition to pyramidal cells. It is unknown whether this circuit motif is a general cortico-cortical feedback organizing principle. Combining multiple simultaneous whole-cell recordings with optogenetics we found that in contrast to this wiring rule, feedback between the hierarchically organized visual areas (lateral-medial to V1) preferentially activated somatostatin-expressing interneurons. Functionally, both feedback circuits temporally sharpened feed-forward excitation by eliciting a transient increase followed by a prolonged decrease in pyramidal firing rate under sustained feed-forward input. However, under feed-forward transient input, the motor-sensory feedback facilitated pyramidal cell bursting while visual feedback increased spike time precision. Our findings argue for multiple feedback motifs implementing different dynamic non-linear operations.

neuroscience↗

bioRxiv2019

A global map of orientation tuning in mouse visual cortex

In primates and most carnivores, neurons in primary visual cortex are spatially organized by their functional properties. For example, neurons with similar orientation preferences are grouped together in iso-orientation domains that smoothly vary over the cortical sheet. In rodents, on the other hand, neurons with different orientation preferences are thought to be spatially intermingled, a feature which has been termed \"salt-and-pepper\" organization. The apparent absence of any systematic structure in orientation tuning has been considered a defining feature of the rodent visual system for more than a decade, with broad implications for brain development, visual processing, and comparative neurophysiology. Here, we revisited this question using new techniques for wide-field two-photon calcium imaging that enabled us to collect nearly complete population tuning preferences in layers 2-4 across a large fraction of the mouse visual hierarchy. Examining the orientation tuning of these hundreds of thousands of neurons, we found a global map spanning multiple visual cortical areas in which orientation bias was organized around a single pinwheel centered in V1. This pattern was consistent across animals and cortical depth. The existence of this global organization in rodents has implications for our understanding of visual processing and the principles governing the ontogeny and phylogeny of the visual cortex of mammals.

neuroscience↗

bioRxiv2019

Cell type composition and circuit organization of neocortical radial clones

SummaryExcitatory neurons arising from a common progenitor establish radially-oriented clonal units in the neocortex which have been proposed to serve as elementary information processing modules. To characterize the cell types and circuit diagram within these clonal units, we performed single-cell RNA-sequencing and multi-cell patch clamp recordings of neurons derived from Nestin-positive progenitors. We found that radial clones do not appear to be fate-restricted, but instead individual clones are composed of a random sampling of the transcriptomic cell types present in a particular cortical area. The effect of lineage on synaptic connectivity depends on the type of connection tested: pairs of clonally related neurons were more likely to be connected vertically, across cortical layers, but not laterally within the same layer, compared to unrelated pairs. We propose that integration of vertical input from related neurons with lateral input from unrelated neurons may represent a developmentally programmed motif for assembling neocortical circuits.

neuroscience↗

bioRxiv2018

Inception in visual cortex: in vivo-silico loops reveal most exciting images

Much of our knowledge about sensory processing in the brain is based on quasi-linear models and the stimuli that optimally drive them. However, sensory information processing is nonlinear, even in primary sensory areas, and optimizing sensory input is difficult due to the high-dimensional input space. We developed inception loops, a closed-loop experimental paradigm that combines in vivo recordings with in silico nonlinear response modeling to identify the Most Exciting Images (MEIs) for neurons in mouse V1. When presented back to the brain, MEIs indeed drove their target cells significantly better than the best stimuli identified by linear models. The MEIs exhibited complex spatial features that deviated from the textbook ideal of V1 as a bank of Gabor filters. Inception loops represent a widely applicable new approach to dissect the neural mechanisms of sensation.

neuroscience↗

bioRxiv2018

Stimulus domain transfer in recurrent models for large scale cortical population prediction on video

To better understand the representations in visual cortex, we need to generate better predictions of neural activity in awake animals presented with their ecological input: natural video. Despite recent advances in models for static images, models for predicting responses to natural video are scarce and standard linear-nonlinear models perform poorly. We developed a new deep recurrent network architecture that predicts inferred spiking activity of thousands of mouse V1 neurons simulta-neously recorded with two-photon microscopy, while accounting for confounding factors such as the animals gaze position and brain state changes related to running state and pupil dilation. Powerful system identification models provide an opportunity to gain insight into cortical functions through in silico experiments that can subsequently be tested in the brain. However, in many cases this approach requires that the model is able to generalize to stimulus statistics that it was not trained on, such as band-limited noise and other parameterized stimuli. We investigated these domain transfer properties in our model and find that our model trained on natural images is able to correctly predict the orientation tuning of neurons in responses to artificial noise stimuli. Finally, we show that we can fully generalize from movies to noise and maintain high predictive performance on both stimulus domains by fine-tuning only the final layers weights on a network otherwise trained on natural movies. The converse, however, is not true.

neuroscience↗