Biology subjects

Xu, H. A.

Publications and source records attributed to Xu, H. A..

2 recordsLinked to original sources

bioRxiv2022

The curse of optimism: a persistent distraction by novelty

Humans successfully explore their environment to find extrinsic rewards, even when exploration requires several intermediate reward-free decisions. It has been hypothesized that intrinsic rewards such as novelty guide this reward-free exploration. However, different intrinsic rewards lead to different exploration strategies, some prone to suboptimal attraction to irrelevant stochastic stimuli, sometimes called the noisy TV problem. Here, we ask whether humans show a similar attraction to reward-free stochasticity and, if so, which type of intrinsic reward guides their exploration. We design a multi-step decision-making paradigm where human participants search for rewarding states in an environment with a highly stochastic but reward-free sub-region. We show that (i) participants persistently explore the stochastic sub-region and (ii) their decisions are best explained by algorithms driven by novelty but not by optimal algorithms driven by information gain. Our results suggest that humans use suboptimal but computationally cheap strategies for exploration in complex environments.

neuroscience↗

bioRxiv2020

Novelty is not Surprise: Exploration and learning in human sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in response to changes in the environment or in the absence of external reward. Here, we design a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning but is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

neuroscience↗