Biology subjects

Serences, J. T.

Publications and source records attributed to Serences, J. T..

15 recordsLinked to original sources

bioRxiv2025

Distributed and drifting signals for working memory load in human cortex

Increasing working memory (WM) load incurs behavioral costs, and whether the neural constraints on behavioral costs are localized (i.e., emanating from the intraparietal sulcus) or distributed across cortex remains an active area of debate. In a pre-registered fMRI experiment, 12 humans (12 scanner-hours each) performed a visual WM task with varying memory load (0-4 items). We replicated a localized, load-dependent increase in univariate BOLD activity in parietal cortex. However, we also observed both systematic increases and decreases in univariate activity with load across the visual hierarchy. Importantly, multivariate activation patterns encoded WM load regardless of the direction of the univariate effect, arguing against a restricted locus of load signals in parietal cortex. Finally, we observed representational drift in activity patterns encoding memory load across scanning sessions. Our results suggest a distributed code for memory load that may be continually refined over time to support more efficient information storage.

neuroscience↗

bioRxiv2024

A 2D Gabor-wavelet baseline model out-performs a 3D surface model in scene-responsive cortex

Understanding 3D representations of spatial information, particularly in naturalistic scenes, remains a significant challenge in vision science. This is largely because of conceptual difficulties in disentangling higher-level 3D information from co-occurring features and cues (e.g., the 3D shape of a scene image is necessarily defined by "low-level" spatial frequency and orientation information). Recent work has employed newer models and analysis techniques that attempt to mitigate these difficulties within a model-comparison framework. For example, one such study reported 3D-surface features were uniquely present in areas OPA, PPA, and MPA/RSC (areas typically referred to as scene-selective), above and beyond a Gabor-wavelet baseline model. Here, we tested whether these findings generalized to a new stimulus set that, on average, dissociated static Gabor-wavelet baseline features from 3D scene-surface features. Surprisingly, we found evidence that a Gabor-wavelet baseline model--commonly thought of as a "low-level" or "2D" model--better fit voxel responses in areas OPA, PPA and MPA/RSC compared to a model with 3D-surface information. We highlight that this difference in results could be due to differences in the baseline conditions used across studies. These findings emphasize that much of the information in "scene-selective" regions--potentially even information about 3D surfaces--may be in the form of spatial frequency and orientation information often considered 2D or low-level. Disentangling lower-level and higher-level visual information is a continuing fundamental challenge for model-comparison approaches in visual cognition, and it motivates future work investigating which visual features could cue higher-level properties in our real-world visual experience--both within and beyond current model comparison frameworks. Author summaryTo gain a more complete picture of human visual processing, it is critical to understand the precise format of representations of naturalistic visual scenes. Recent work has approached this challenge by quantifying how much of our brain activity might be due to hypothesized characteristics of the stimuli being viewed. Here, we followed up on work finding that activity in scene-responsive regions of the brain is well predicted by information about the 3D configurations of major surfaces in viewed scenes, like walls and floors. In contrast to previous work, we found that our baseline condition--commonly thought of as "low-level" visual information--accounted for responses in these regions better than both of the 3D surface models that we tested. We highlight that this difference in results could be due to differences in the baseline conditions used across studies. However, our findings do not necessarily argue against the importance of these regions in encoding 3D surface information. Instead, they highlight the possibility that baseline models can perform well by virtue of covariation between low-level features like orientation and spatial frequency with higher-level properties like depth information. This motivates future work and/or new analysis frameworks to better characterize the interplay between hypothesized model features and the specific techniques used to quantify their relationship to neural representations.

neuroscience↗

bioRxiv2023

Categorization dynamically alters representations in human visual cortex

Everyday perceptual tasks require sensory stimuli to be dynamically encoded and analyzed according to changing behavioral goals. For example, when searching for an apple at the supermarket, one might first find the Granny Smith apples by separating all visible apples into the categories "green" and "non-green". However, suddenly remembering that your family actually likes Fuji apples would necessitate reconfiguring the boundary to separate "red" from "red-yellow" objects. This flexible processing enables identical sensory stimuli to elicit varied behaviors based on the current task context. While this phenomenon is ubiquitous in nature, little is known about the neural mechanisms that underlie such flexible computation. Traditionally, sensory regions have been viewed as mainly devoted to processing inputs, with limited involvement in adapting to varying task contexts. However, from the standpoint of efficient computation, it is plausible that sensory regions integrate inputs with current task goals, facilitating more effective information relay to higher-level cortical areas. Here we test this possibility by asking human participants to visually categorize novel shape stimuli based on different linear and non-linear boundaries. Using fMRI and multivariate analyses of retinotopically-defined visual areas, we found that shape representations in visual cortex became more distinct across relevant decision boundaries in a context-dependent manner, with the largest changes in discriminability observed for stimuli near the decision boundary. Importantly, these context-driven modulations were associated with improved categorization performance. Together, these findings demonstrate that codes in visual cortex are adaptively modulated to optimize object separability based on currently relevant decision boundaries.

neuroscience↗

bioRxiv2023

Distinguishing response from stimulus driven history biases

Perception is shaped by past experience, both cumulative and contextual. Serial dependence reflects a contextual attractive bias to perceive or report the world as more stable than it truly is. As serial dependence has often been examined in continuous report or change detection tasks, it unclear whether attraction is towards the identity of the previous stimulus feature, or rather to the response made to indicate the perceived stimulus value on the previous trial. The physical and reported identities can be highly correlated depending on properties of the stimulus and task design. However, they are distinct values and dissociating them is important because it can reveal information about the role of sensory and non-sensory contributions to attractive biases. These alternative possibilities can be challenging to disentangle because 1) stimulus values and responses are typically strongly correlated and 2) measuring response biases using standard techniques can be confounded by context-independent biases such as cardinal bias for orientation (i.e., higher precision, but repelled, responses from vertical and horizontal orientations). Here we explore the issues and confounds related to measuring response biases using simulations. Under a range of conditions, we find that response-induced biases can be reliably distinguished from stimulus-induced biases and from confounds introduced by context-independent biases. We then applied these approaches to a delayed report dataset (N=18) and found evidence for response over a stimulus driven history bias. This work demonstrates that stimulus and response driven history biases can be reliably dissociated and provides code to implement these analysis procedures.

neuroscience↗

bioRxiv2021

Flexible utilization of spatial- and motor-based codes for the storage of visuo-spatial information

Working memory provides flexible storage of information in service of upcoming behavioral goals. Some models propose specific fixed loci and mechanisms for the storage of visual information in working memory, such as sustained spiking in parietal and prefrontal cortex during working memory maintenance. An alternative view is that information can be remembered in a flexible format that best suits current behavioral goals. For example, remembered visual information might be stored in sensory areas for easier comparison to future sensory inputs, or might be re-coded into a more abstract action-oriented format and stored in motor areas. Here, we tested this hypothesis using a visuo-spatial working memory task where the required behavioral response was either known or unknown during the memory delay period. Using fMRI and multivariate decoding, we found that there was less information about remembered spatial position in early visual and parietal regions when the required response was known versus unknown. Further, a representation of the planned motor action emerged in primary somatosensory, primary motor, and premotor cortex during the same task condition where spatial information was reduced in early visual cortex. These results suggest that the neural networks supporting working memory can be strategically reconfigured depending on specific behavioral requirements during a canonical visual working memory paradigm.

neuroscience↗

bioRxiv2021

Sensory readout accounts for adaptation

Sensory responses and behavior are strongly shaped by stimulus history. For instance, perceptual reports are sometimes biased towards previously viewed stimuli (serial dependence). While behavioral studies have pointed to both perceptual and post-perceptual origins of this phenomenon, neural data that could elucidate where these biases emerge is limited. We recorded fMRI responses while human participants (male and female) performed a delayed orientation discrimination task. While behavioral reports were attracted to the previous stimulus, response patterns in visual cortex were repelled. We reconciled these opposing neural and behavioral biases using a model where both sensory encoding and readout are shaped by stimulus history. First, neural adaptation reduces redundancy at encoding and leads to the repulsive biases that we observed in visual cortex. Second, our modeling work suggest that serial dependence is induced by readout mechanisms that account for adaptation in visual cortex. According to this account, the visual system can simultaneously improve efficiency via adaptation while still optimizing behavior based on the temporal structure of natural stimuli.

neuroscience↗

bioRxiv2021

Shared long-term and short-term memory representational formats in occipital and parietal cortex

Current theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. Both are thought to rely on reinstating patterns of sensory activity evoked by the perception of the remembered item. However, the extent of the shared representations between WM and LTM are unclear, and it is unknown how WM and LTM representations may differ across cortical regions. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM. Critically, we carefully matched the precision of behavioral responses in these tasks. We used fMRI and multivariate pattern analyses to examine representations in (1) retinotopic cortex and (2) lateral parietal cortex (LPC) regions previously implicated in LTM. We show that visual memories were represented in a sensory-like code in both tasks across retinotopic regions in occipital and parietal cortex. LPC regions also encoded remembered locations in both WM and LTM, but in a format that differed from the sensory-evoked activity. These results suggest a striking correspondence in the format of WM and LTM representations across occipital and parietal cortex. On the other hand, we show that activity patterns in nearly all parietal regions, but not occipital regions, contained information that could discriminate between WM trials and LTM trials. Our data provide new evidence for theories of memory systems and the representation of mnemonic content.

neuroscience↗

bioRxiv2020

History-driven modulations of population codes in early visual cortex during visual search

To find important objects, we must focus on our goals, ignore distractions, and take our changing environment into account. This is formalized in models of visual search whereby goal-driven, stimulus-driven and history-driven factors are integrated into a priority map that guides attention. Stimulus history robustly influences where attention is allocated even when the physical stimulus is the same: when a salient distractor is repeated over time, it captures attention less effectively. A key open question is how we come to ignore salient distractors when they are repeated. Goal-driven accounts propose that we use an active, expectation-driven mechanism to attenuate the distractor signal (e.g., predictive coding), whereas stimulus-driven accounts propose that the distractor signal is attenuated due to passive changes to neural activity and inter-item competition (e.g., adaptation). To test these competing accounts, we measured item-specific fMRI responses in human visual cortex during a visual search task where trial history was manipulated (colors unpredictably switched or were repeated). Consistent with a stimulus-driven account of history-based distractor suppression, we found that repeated singleton distractors were suppressed starting in V1, and distractor suppression did not increase in later visual areas. In contrast, we observed signatures of goal-driven target enhancement that were absent in V1, increased across visual areas, and were not modulated by stimulus history. Our data suggest that stimulus history does not alter goal-driven expectations, but rather modulates canonically stimulus-driven sensory responses to contribute to a temporally-integrated representation of priority. Significance StatementVisual search refers to our ability to find what we are looking for in a cluttered visual world (e.g., finding your keys). To perform visual search, we must integrate information about our goals (e.g., find the red key-chain), the environment (e.g., salient items capture your attention), and changes to the environment (i.e., stimulus history). Although stimulus history impacts behavior, the neural mechanisms that mediate history-driven effects remain debated. Here, we leveraged fMRI and multivariate analysis techniques to measure history-driven changes to the neural representation of items during visual search. We found that stimulus history influenced the representation of a salient pop-out distractor starting in V1, suggesting that stimulus history operates via modulations in early sensory processing rather than goal-driven expectations.

neuroscience↗

bioRxiv2018

Simultaneous representation of sensory and mnemonic information in human visual cortex

Traversing sensory environments requires keeping relevant information in mind while simultaneously processing new inputs. Visual information is kept in working memory via feature selective responses in early visual cortex, but recent work had suggested that new sensory inputs wipe out this information. Here we show region-wide multiplexing abilities in classic sensory areas, with population-level response patterns in visual cortex representing the contents of working memory concurrently with new sensory inputs.

neuroscience↗

bioRxiv2018

Alpha-band oscillations track the retrieval of precise spatial representations from long-term memory

A hallmark of episodic memory is the phenomenon of mentally re-experiencing the details of past events, and a well-established concept is that the neuronal activity that mediates encoding is reinstated at retrieval. Evidence for reinstatement has come from multiple modalities, including functional Magnetic Resonance Imaging (fMRI) and electroencephalography (EEG). These EEG studies have shed light on the time-course of reinstatement, but have been limited to distinguishing between a few categories and/or limited measures of memory strength. The goal of this work was to investigate whether recently developed experimental and technical approaches, namely an inverted encoding model applied to alpha oscillatory power in conjunction with sensitive tests of memory retrieval in a continuous space, can track and reconstruct memory retrieval of specific spatial locations. In Experiment 1, we establish that an inverted encoding model applied to multivariate alpha topography can track retrieval of precise spatial memories. In Experiment 2, we demonstrate that the pattern of multivariate alpha activity at study is similar to the pattern observed during retrieval. Finally, we observe that these encoding models predict memory retrieval behavior, including the accuracy and latency of recall. These findings highlight the broad potential for using encoding models to characterize long-term memory retrieval.

neuroscience↗

bioRxiv2017

Dissociable signatures of visual salience and behavioral relevance across attentional priority maps in human cortex

Computational models posit that visual attention is guided by activity within spatial maps that index the image-computable salience and the behavioral relevance of objects in the scene. However, the simultaneous influence of these factors on putative neural attentional priority maps in human cortex is not well understood. We tested the hypothesis that visual salience and behavioral relevance independently impact the activation profile across retinotopically-organized cortical regions by quantifying attentional priority maps measured in human brains using functional MRI while participants attended one of two differentially-salient stimuli. We find that the topography of activation in priority maps, as reflected in the modulation of region-level patterns of population activity, independently indexed the physical salience and behavioral relevance of each scene element. Moreover, salience strongly impacted activation patterns in early visual areas, whereas later visual areas were dominated by relevance. This suggests that prioritizing spatial locations relies on distributed neural codes containing graded representations of salience and relevance across the visual hierarchy.\n\nSignificance StatementOften, it is necessary to orient towards bright, unique, or sudden events in the environment - that is, salient stimuli. However, we can focus processing resources on less salient visual information if it is relevant to the task at hand. We tested a theory which supposes that we represent different scene elements according to both their salience and their relevance in a series of priority maps by measuring fMRI activation patterns across the human brain and reconstructing spatial maps of the visual scene under different task conditions. We found that different regions indexed either the salience or the relevance of scene items, but not their interaction, suggesting an evolving representation of salience and relevance across different visual areas.

neuroscience↗

bioRxiv2017

Category learning biases sensory representations in visual cortex.

Categorization allows organisms to generalize existing knowledge to novel stimuli and to discriminate between physically similar yet conceptually different stimuli. Humans, nonhuman primates, and rodents can readily learn arbitrary categories defined by low-level visual features, and learning distorts perceptual sensitivity for category-defining features such that differences between physically similar yet categorically distinct exemplars are enhanced while differences between equally similar but categorically identical stimuli are reduced. We report a basis for these distortions in human occipitoparietal cortex. In three experiments, we used an inverted encoding model to recover population-level representations of stimuli from multivoxel and multi-electrode patterns of human brain activity while human participants (both sexes) classified continuous stimulus sets into discrete groups. In each experiment, reconstructed representations of to-be-categorized stimuli were systematically biased towards the center of the appropriate category. These biases were largest for exemplars near a category boundary, predicted participants overt category judgments, emerged shortly after stimulus onset, and could not be explained by mechanisms of response selection or motor preparation. Collectively, our findings suggest that category learning can influence processing at the earliest stages of cortical visual processing. Significance StatementCategory learning enhances perceptual sensitivity for physically similar yet categorically different stimuli. We report a possible mechanism for these distortions in human occipitoparietal cortex.. In three experiments, we used an inverted encoding model to recover population-level representations of stimuli from multivariate patterns in occipitoparietal cortex while participants categorized sets of continuous stimuli into discrete groups. The recovered representations were systematically biased by category membership, with larger biases for exemplars adjacent to a category boundary. These results suggest that mechanisms of categorization shape information processing at the earliest stages of the visual system.

neuroscience↗

bioRxiv2016

Training qualitatively shifts the neural mechanisms that support attentional selection

Attention supports the selection of relevant sensory information from competing irrelevant sensory information. This selective processing is thought to be supported via the attentional gain amplification of sensory responses evoked by attended compared to unattended stimuli. However, recent studies in highly trained subjects suggest that attentional gain plays a relatively modest role and that other types of neural modulations - such as a reduction in neural noise - better explain attention-related changes in behavior. We hypothesized that the amount of training may alter neural mechanisms that support attentional selection in visual cortex. To test this hypothesis, we investigated the influence of training on attentional modulations of stimulus-evoked visual responses by recording electroencephalography (EEG) from humans performing a selective visuospatial attention task over the course of one month. Early in training, visuospatial attention induced a robust attentional gain amplification of sensory-evoked responses in contralateral visual cortex that emerged within ~100ms after stimulus onset, and a quantitative model based on signal detection theory (SDT) successfully linked this attentional gain amplification to attention-related improvements in behavior. However, after training, this attentional gain amplification of visual responses was almost completely eliminated and modeling suggested that noise reduction was required to link the amplitude of visual responses with attentional modulations of behavior. These findings suggest that the neural mechanisms supporting selective attention can change as a function of training and expertise, and help to bridge different results from studies carried out in different model systems that require substantially different amount of training.

neuroscience↗

bioRxiv2016

Stimulus visibility controls the balance between attention-induced changes in contrast appearance and decision bias

While attention is known to improve information processing, whether attention can alter visual appearance has been a cornerstone of debate for 100+ years. Although recent studies suggest that attention can alter appearance, it has been argued that the reported appearance changes reflect response bias. Here, we provide a resolution to this debate by showing that attention has different effects on appearance and response bias depending on stimulus visibility. In a contrast judgment task where the contrast of attended and unattended stimuli varied across a full range of contrast values, human participants exhibited a substantial amount of response bias to the attended stimulus, when stimuli were hard to see. However, when stimuli were easier to see, response bias decreased and attention primarily increased perceived contrast. These results help constrain philosophical arguments about the cognitive penetrability of perception and reconcile the long-standing debate about the attention effect on appearance and response bias.

neuroscience↗

bioRxiv2016

Feature-coding transitions to conjunction-coding with progression through human visual cortex

Identifying an object and distinguishing it from similar items depends upon the ability to perceive its component parts as conjoined into a cohesive whole, but the brain mechanisms underlying this ability remain elusive. The ventral visual processing pathway in primates is organized hierarchically: Neuronal responses in its early stages are sensitive to the manipulation of simple visual features whereas neuronal responses in subsequent stages are tuned to increasingly complex stimulus attributes. It is widely assumed that feature-coding dominates in early visual cortex whereas later visual regions employ conjunction-coding in which object representations are different from the sum of their simple-feature parts. However, no study has demonstrated that putative object-level codes in higher visual cortex cannot be accounted for by feature-coding and that putative feature-codes in regions prior to ventral temporal cortex are not equally well characterized as object-level codes. Thus the existence of a transition from feature- to conjunction-coding in visual cortex remains unconfirmed, and, if a transition does occur, its location remains unknown. By employing multivariate analysis of human functional imaging data, we measure both feature-coding and conjunction-coding directly, using the same set of visual stimuli, and pit them against each other to reveal the relative dominance of one versus the other throughout cortex. We provide the first demonstration of a transition from feature-coding in early visual cortex to conjunction-coding in both inferior temporal and posterior parietal cortices. This novel method enables the use of experimentally controlled stimulus features to investigate population-level feature- and conjunction-codes throughout human cortex.

neuroscience↗