Biology subjects

Tuzsus, D.

Publications and source records attributed to Tuzsus, D..

8 recordsLinked to original sources

bioRxiv2026

Individual Differences in Dopaminergic Modulation of Exploration-Exploitation Behaviour

Dopamine (DA) has been implicated in exploration-exploitation behaviour, i.e., exploring novel, potentially better options vs. exploiting known, previously rewarding options. Impairments in this trade-off occur in psychiatric disorders involving DAergic dysfunction, including addiction and schizophrenia. Pharmacological studies revealed a contribution of DA to exploration, but inconsistent findings suggest that interindividual variability in baseline DA may modulate effects. To address this, we investigated the effects of the DA precursor L-DOPA on exploration-exploitation during reinforcement learning in a sample of N = 75 healthy participants (n = 32 women), following a randomised, double-blind, placebo-controlled, pre-registered design (https://osf.io/p2r7u). We assessed whether putative baseline DA markers, including spontaneous eye blink rate, working memory (WM) capacity, and impulsivity, modulated drug effects and probed visual fixation patterns and pupil dilation as markers of exploration. L-DOPA had no overall effect on computational model parameters of random exploration, directed exploration or choice perseveration. WM capacity moderated drug effects on random exploration, with stronger effects at higher WM capacity. Remaining DA proxies showed no credible effects. Pooling the data from male participants with that from an earlier male-only study (Chakroun et al., 2020; total N = 74), L-DOPA increased uncertainty-dependent value weighting and perseveration strength, while decreasing habit updating, indicating a stronger tendency to repeat previous choices and slower decay of their influence over time. No credible drug effects were observed in female participants. Pupil dilation was tonically increased under L-DOPA and scaled with exploration behaviour and prediction error, confirming that pupillometry can index exploration-exploitation dynamics. Visual exploration patterns reflected uncertainty-driven sampling, but were unaffected by L-DOPA. Taken together, results suggest that DAergic modulation of exploration and perseveration behaviour may be contingent on cognitive capacity and sex, rather than exerting uniform effects across individuals.

neuroscience↗

bioRxiv2025

Recurrent Neural Network Exploration Strategies During Reinforcement Learning Depend on Network Capacity

Artificial neural networks constitute simplified computational models of neural circuits that might help understand how the biological brain solves and represents complex tasks. Previous research revealed that recurrent neural networks (RNNs) with 48 hidden units show human-level performance in restless four-armed bandit tasks but differ from humans with respect to the task strategy employed. Here we systematically examined the impact of network capacity (no. of hidden units) on computational mechanisms and performance. Computational modeling was applied to investigate and compare network behavior between capacity levels as well as between RNNs and human learners. Using a task frequently employed in human cognitive neuroscience work as well as in animal systems neuroscience work, we show that high-capacity networks displayed increased directed exploration and attenuated random exploration relative to low-capacity networks. RNNs with 576 hidden units approached "human-like" exploration strategies, but the overall switch rate and the level of perseveration still deviated from human learners. In the context of the resource-rational framework, which posits a trade-off between reward and policy complexity, human learners may devote more resources to solving the task, albeit without performance benefits over RNNs. Taken together, this work reveals the importance of network capacity on exploration strategies during reinforcement learning and therefore contributes to the goal of building neural networks that behave "human-like" to possibly gain insights into computational mechanisms in human brains.

animal behavior and cognition↗

bioRxiv2024

Dopamine and temporal discounting: revisiting pharmacology and individual differences

Disorders characterised by changes in dopamine (DA) neurotransmission are often linked to changes in the temporal discounting of future rewards. Likewise, pharmacological manipulations of DA neuro-transmission in healthy individuals modulate temporal discounting, but there is considerable variability in the directionality of reported pharmacological effects, as enhancements and reductions of DA signalling have been linked to both increases and reductions of temporal discounting. This may be due to meaningful individual differences in drug effects and/or false positive findings in small samples. To resolve these inconsistencies, we 1) revisited pharmacological effects of the DA precursor L-DOPA on temporal discounting in a large sample of N = 76 healthy participants (n = 44 male) and 2) examined several putative proxy measures for DA to revisit the role of individual differences in a randomised, double-blind placebo-controlled pre-registered study (https://osf.io/a4k9j/). Replicating previous findings, higher rewards were discounted less (magnitude effect). Computational modelling using hierarchical Bayesian parameter estimation confirmed that the data in both drug conditions were best accounted for by a non-linear temporal discounting drift diffusion model. In line with recent animal and human work, L-DOPA reliably reduced the discount rate with a small effect size, challenging earlier findings in substantially smaller samples. We found no credible evidence for linear or quadratic effects of putative DA proxy measures on model parameters, calling into question the role of these measures in accounting for individual differences in DA drug effects.

neuroscience↗

bioRxiv2024

A meta reinforcement learning account of behavioral adaptation to volatility in recurrent neural networks

Natural environments often exhibit various degrees of volatility, ranging from slowly changing to rapidly changing contingencies. How learners adapt to changing environments is a central issue in both reinforcement learning theory and psychology. For example, learners may adapt to changes in volatility by increasing learning if volatility increases, and reducing it if volatility decreases. Computational neuroscience and neural network modeling work suggests that this adaptive capacity may result from a meta-reinforcement learning process (implemented for example in the prefrontal cortex), where past experience endows the system with the capacity to rapidly adapt to environmental conditions in new situations. Here we provide direct evidence for a meta-reinforcement learning account of adaptation to environmental volatility. Recurrent neural networks (RNNs) were trained on a restless four-armed bandit reinforcement learning problem under three different training regimes (low volatility training only, medium volatility training only, or meta-volatility training across a range of volatilities). We show that, in contrast to RNNs trained in the low volatility regime, RNNs trained under the medium or meta-volatility regimes exhibited a superior adaption to environmental volatility. This was reflected in a better performance, and computational modeling of the networks behavior revealed a more adaptive adjustment of learning and exploration to varying levels of volatility during test. Results extend the meta-RL account to volatility adaptation and confirm past experience as a crucial factor of adaptive decision-making.

neuroscience↗

bioRxiv2023

Pharmacological enhancement of dopamine neurotransmission does not affect illusory pattern perception

Psychotic symptoms and delusional belief have been linked to dopamine transmission in both healthy and clinical samples and are assumed to result at least in part from perceiving illusory patterns in noise. However, the existing literature on the role of dopamine in detecting patterns in noise is inconclusive. To address this issue, we assessed the effect of manipulating dopaminergic neurotransmission on illusory pattern perception in healthy individuals (n = 48, n = 19 female) in a double-blind placebo-controlled within-subjects design (see preregistration at https://osf.io/a4k9j/). We predicted individuals on vs. off L-DOPA to be more likely to perceive illusory patterns, specifically objects in images containing only noise. Using a signal detection model, however, we found no credible evidence that L-DOPA compared to placebo increased false alarm rates. Further, L-DOPA did not modulate measures of accuracy, discrimination sensitivity and response bias. In all cases, Bayesian statistics revealed strong evidence in favour of the null hypothesis Future studies should address possible dose-dependent effects and differential effects in healthy vs. clinical samples.

neuroscience↗

bioRxiv2023

Human-level reinforcement learning performance of recurrent neural networks is linked to hyperperseveration, not directed exploration

A key feature of animal and human decision-making is to balance the exploration of unknown options for information gain (directed exploration) versus selecting known options for immediate reward (exploitation), which is often examined using restless bandit tasks. Recurrent neural network models (RNNs) have recently gained traction in both human and systems neuroscience work on reinforcement learning, due to their ability to show meta-learning of task domains. Here we comprehensively compared the performance of a range of RNN architectures as well as human learners on restless four-armed bandit problems. The best-performing architecture (LSTM network with computation noise) exhibited human-level performance. Computational modeling of behavior first revealed that both human and RNN behavioral data contain signatures of higher-order perseveration, i.e., perseveration beyond the last trial, but this effect was more pronounced in RNNs. In contrast, human learners, but not RNNs, exhibited a positive effect of uncertainty on choice probability (directed exploration). RNN hidden unit dynamics revealed that exploratory choices were associated with a disruption of choice predictive signals during states of low state value, resembling a win-stay-loose-shift strategy, and resonating with previous single unit recording findings in monkey prefrontal cortex. Our results highlight both similarities and differences between exploration behavior as it emerges in meta-learning RNNs, and computational mechanisms identified in cognitive and systems neuroscience work.

neuroscience↗

bioRxiv2022

Erotic cue exposure increases physiological arousal, biases choices towards immediate rewards and attenuates model-based reinforcement learning

Computational psychiatry focuses on identifying core cognitive processes that appear altered across a broad range of psychiatric disorders. Temporal discounting of future rewards and model-based control during reinforcement learning have proven as two promising candidates. Despite its trait-like stability, temporal discounting has been suggested to be at least partly under contextual control. For example, highly arousing cues such as erotic pictures were shown to increase discounting, although overall evidence to date remains somewhat mixed. Whether model-based reinforcement learning is similarly affected by arousing cues is unclear. Here we tested cue-reactivity effects (erotic pictures) on subsequent temporal discounting and model-based reinforcement learning in a within-subjects design in n=39 healthy male participants. Self-reported and physiological arousal (cardiac activity and pupil dilation) were assessed before and during cue exposure. Arousal was increased during exposure of erotic vs. neutral cues both on the subjective and autonomic level. Erotic cue exposure nominally increased discounting as reflected by reduced choices of delayed options. Hierarchical drift diffusion modeling (DDM) linked increased discounting to a shift in the starting point bias of evidence accumulation towards immediate options. Model-based control during reinforcement learning was reduced following erotic cues according to model-agnostic analysis. Notably, DDM linked this effect to attenuated forgetting rates of unchosen options, leaving the model-based control parameter unchanged. Our findings replicate previous work on cue-reactivity effects in temporal discounting and for the first time show similar effects in model-based reinforcement learning. Our results highlight how environmental cues can impact core human decision processes and reveal that comprehensive drift diffusion modeling approaches can yield novel insights in reward-based decision processes.

neuroscience↗

bioRxiv2021

Parameter and model recovery of reinforcement learning models for restless bandit problems

Multi-armed restless bandit tasks are regularly applied in psychology and cognitive neuroscience to assess exploration and exploitation behavior in structured environments. These models are also readily applied to examine effects of (virtual) brain lesions on performance, and to infer neurocomputational mechanisms using neuroimaging or pharmacological approaches. However, to infer individual, psychologically meaningful parameters from such data, computational cognitive modeling is typically applied. Recent studies indicate that softmax (SM) decision rule models that include a representation of environmental dynamics (e.g. the Kalman Filter) and additional parameters for modeling exploration and perseveration (Kalman SMEP) fit human bandit task data better than competing models. Parameter and model recovery are two central requirements for computational models: parameter recovery refers to the ability to recover true data-generating parameters; model recovery refers to the ability to correctly identify the true data generating model using model comparison techniques. Here we comprehensively examined parameter and model recovery of the Kalman SMEP model as well as nested model versions, i.e. models without the additional parameters, using simulation and Bayesian inference. Parameter recovery improved with increasing trial numbers, from around .8 for 100 trials to around .93 for 300 trials. Model recovery analyses likewise confirmed acceptable recovery of the Kalman SMEP model. Model recovery was lower for nested Kalman filter models as well as delta rule models with fixed learning rates. Exploratory analyses examined associations of model parameters with model-free performance metrics. Random exploration, captured by the inverse softmax temperature, was associated with lower accuracy and more switches. For the exploration bonus parameter modeling directed exploration, we confirmed an inverse-U-shaped association with accuracy, such that both an excess and a lack of directed exploration reduced accuracy. Taken together, these analyses underline that the Kalman SMEP model fulfills basic requirements of a cognitive model.

neuroscience↗