Biology subjects

Gershman, S.

Publications and source records attributed to Gershman, S..

9 recordsLinked to original sources

bioRxiv2026

Error-driven representation learning in the mesolimbic system

In reinforcement learning, an agent learns to map representations of the environment state to predictions of future reward. Most prior work in neuroscience has assumed a fixed representation and studied how reward prediction errors (thought to be conveyed by phasic dopamine signals) are used to update the mapping from representations to predictions. However, work in machine learning has demonstrated that much more powerful predictive systems can be learned by using the errors to update the representations themselves. We study whether the brain does something similar by leveraging simultaneous recordings of striatal projection neurons in the olfactory tubercle (putatively representing state features) and dopamine neurons in the ventral tegmental area. We show that trial-by-trial changes in striatal activity are more consistent with dopamine-driven representation learning than a variety of alternative updating schemes. This result suggests a convergence of representation learning principles in biological and artificial systems.

neuroscience↗

bioRxiv2020

A low power flexible dielectric barrier discharge disinfects surfaces and improves the action of hydrogen peroxide

There is an urgent need for disinfection and sterilization devices accessible to the public that can be fulfilled by innovative strategies for using cold atmospheric pressure plasmas. Here we explore and evaluate an approach combining a flexible printed circuit design of a dielectric barrier discharge (DBD) with an environmentally safe chemical reagent for surface decontamination from bacterial contaminants. The device operates in ambient air without any additional gas flow at a power density of about 0.5 W/cm2. Using a 3% hydrogen peroxide solution as a model reagent we demonstrate improved reduction in the bacterial load of a surface contaminant of >6log10 in 90 seconds, about 3log10 and 2log10 faster compared to hydrogen peroxide alone and the flexible DBD alone, respectively, for the same treatment time. The synergistic action of the plasma bioactive properties and hydrogen peroxide result in a dramatic improvement of surface disinfection. This opens new possibilities for using the low power flexible DBD plasma sources for surface disinfection and decontamination.

bioengineering↗

bioRxiv2018

Discovery of Hierarchical Representations for Efficient Planning

We propose that humans spontaneously organize environments into clusters of states that support hierarchical planning, enabling them to tackle challenging problems by breaking them down into sub-problems at various levels of abstraction. People constantly rely on such hierarchical presentations to accomplish tasks big and small - from planning ones day, to organizing a wedding, to getting a PhD - often succeeding on the very first attempt. We formalize a Bayesian model of hierarchy discovery that explains how humans discover such useful abstractions. Building on principles developed in the robotics literature, the model predicts that hierarchy discovery should be sensitive to the topological structure, reward distribution, and distribution of tasks in the environment. In five simulations, we show that the model accounts for previously reported effects of environment structure on planning behavior, such as detection of bottleneck states and transitions. We then test the novel predictions of the model in seven behavioral experiments, demonstrating how the distribution of tasks and rewards can additionally influence planning behavior via the discovered hierarchy, sometimes facilitating and sometimes hindering performance. We also show that people use uncertainty to guide their learning in a way that is informative for hierarchy discovery. Finally, we propose how hierarchy discovery and hierarchical planning might be implemented in the brain. Together, these findings present an important advance in our understanding of how the brain might use Bayesian inference to uncover and exploit the hidden hierarchical structure of the world.

neuroscience↗

bioRxiv2018

Dissociable Neural Correlates of Uncertainty Underlie Different Exploration Strategies

Most real-world decisions involve a delicate balance between exploring unfamiliar alternatives and committing to the best known option. Uncertainty lies at the core of this "explore-exploit" dilemma, for if all options were perfectly known, there would be no need to explore. Yet despite the prominent role of uncertainty-guided exploration in decision making, evidence for its neural implementation is still sparse. We investigated this question with model-based fMRI (n = 31) using a two-armed bandit task that independently manipulates two forms of uncertainty underlying different exploration strategies. The relative uncertainty between the two options was correlated with BOLD activity in right rostrolateral prefrontal cortex and drove directed exploration, a strategy that adds an uncertainty bonus to each option. The total uncertainty across the two options was correlated with activity in left dorsolateral prefrontal cortex and drove random exploration, a strategy that increases choice stochasticity in proportion to total uncertainty. The subjective estimates of uncertainty from both regions were predictive of subject choices. These results are consistent with a hybrid computational architecture in which different uncertainty computations are performed separately and then combined by downstream decision circuits to compute choice.

neuroscience↗

bioRxiv2018

Communicating compositional patterns

How do people perceive and communicate structure? We investigate this question by letting participants play a communication game, where one player describes a pattern, and another player redraws it based on the description alone. We use this paradigm to compare two models of pattern description, one compositional (complex structures built out of simpler ones) and one non-compositional. We find that compositional patterns are communicated more effectively than non-compositional patterns, that a compositional model of pattern description predicts which patterns are harder to describe, and that this model can be used to evaluate participants drawings, producing human-like quality ratings. Our results suggest that natural language can tap into a compositionally structured pattern description language.

animal behavior and cognition↗

bioRxiv2018

Dopaminergic genes are associated with both directed and random exploration

In order to maximize long-term rewards, agents must balance exploitation (choosing the option with the highest payoff) and exploration (gathering information about options that might have higher payoffs). Although the optimal solution to this trade-off is intractable, humans make use of two effective strategies: selectively exploring options with high uncertainty (directed exploration), and increasing the randomness of their choices when they are more uncertain (random exploration). Using a task that independently manipulates these two forms of exploration, we show that single nucleotide polymorphisms related to dopamine are associated with individual differences in exploration strategies. Variation in a gene linked to prefrontal dopamine (COMT) predicted the degree of directed exploration, as well as the overall randomness of responding. Variation in a gene linked to striatal dopamine (DARPP-32) predicted the degree of both directed and random exploration. These findings suggest that dopamine makes multiple contributions to exploration, depending on its afferent target.

neuroscience↗

bioRxiv2018

Reference-dependent preferences arise from structure learning

Modern theories of decision making emphasize the reference-dependency of decision making under risk. In particular, people tend to be risk-averse for outcomes greater than their reference point, and risk-seeking for outcomes less than their reference point. A key question is where reference points come from. A common assumption is that reference points correspond to expectations about outcomes, but it is unclear whether people rely on a single global expectation, or multiple local expectations. If the latter, how do people determine which expectation to apply in a particular situation? We argue that people discover reference points using a form of Bayesian structure learning, which partitions outcomes into distinct contexts, each with its own reference point corresponding to the expected outcome in that context. Consistent with this theory, we show experimentally that dramatic change in the distribution of outcomes can induce the discovery of a new reference point, with systematic effects on risk preferences. By contrast, when changes are gradual, a single reference point is continuously updated.

animal behavior and cognition↗

bioRxiv2017

Neural Computations Underlying Causal Structure Learning

Behavioral evidence suggests that beliefs about causal structure constrain associative learning, determining which stimuli can enter into association, as well as the functional form of that association. Bayesian learning theory provides one mechanism by which structural beliefs can be acquired from experience, but the neural basis of this mechanism is unknown. A recent study (Gershman, 2017) proposed a unified account of the elusive role of \"context\" in animal learning based on Bayesian updating of beliefs about the structure of causal relationships between contexts and cues in the environment. The model predicts that the computations which arbitrate between these abstract causal structures are distinct from the computations which learn the associations between particular stimuli under a given structure. In this study, we used fMRI with male and female human subjects to interrogate the neural correlates of these two distinct forms of learning. We show that structure learning signals are encoded in rostrolateral prefrontal cortex and the angular gyrus, anatomically distinct from correlates of associative learning. Within-subject variability in the encoding of these learning signals predicted variability in behavioral performance. Moreover, representational similarity analysis suggests that some regions involved in both forms of learning, such as parts of the inferior frontal gyrus, may also encode the full probability distribution over causal structures. These results provide evidence for a neural architecture in which structure learning guides the formation of associations.\n\nSignificance StatementAnimals are able to infer the hidden structure behind causal relations between stimuli in the environment, allowing them to generalize this knowledge to stimuli they have never experienced before. A recently published computational model based on this idea provided a parsimonious account of a wide range of phenomena reported in the animal learning literature, suggesting that the neural mechanisms dedicated to learning this hidden structure are distinct from those dedicated to acquiring particular associations between stimuli. Here we validate this model by measuring brain activity during a task which dissociates structure learning from associative learning. We show that different brain networks underlie the two forms of learning and that the neural signal corresponding to structure learning predicts future behavioral performance.

neuroscience↗

bioRxiv2017

Amortized Hypothesis Generation

Bayesian models of cognition posit that people compute probability distributions over hypotheses, possibly by constructing a sample-based approximation. Since people encounter many closely related distributions, a computationally efficient strategy is to selectively reuse computations - either the samples themselves or some summary statistic. We refer to these reuse strategies as amortized inference. In two experiments, we present evidence consistent with amortization. When sequentially answering two related queries about natural scenes, we show that answers to the second query vary systematically depending on the structure of the first query. Using a cognitive load manipulation, we find evidence that people cache summary statistics rather than raw sample sets. These results enrich our notions of how the brain approximates probabilistic inference.

animal behavior and cognition↗