Biology subjects

Tano, P.

Publications and source records attributed to Tano, P..

3 recordsLinked to original sources

bioRxiv2024

Acquiring musculoskeletal skills with curriculum-based reinforcement learning

Efficient musculoskeletal simulators and powerful learning algorithms provide computational tools to tackle the grand challenge of understanding biological motor control. Our winning solution for the inaugural NeurIPS MyoChallenge leverages an approach mirroring human skill learning. Using a novel curriculum learning approach, we trained a recurrent neural network to control a realistic model of the human hand with 39 muscles to rotate two Baoding balls in the palm of the hand. In agreement with data from human subjects, the policy uncovers a small number of kinematic synergies even though it is not explicitly biased towards low-dimensional solutions. However, by selectively inactivating parts of the control signal, we found that more dimensions contribute to the task performance than suggested by traditional synergy analysis. Overall, our work illustrates the emerging possibilities at the interface of muscu-loskeletal physics engines, reinforcement learning and neuro-science to advance our understanding of biological motor control.

neuroscience↗

bioRxiv2023

Multi-timescale reinforcement learning in the brain

To thrive in complex environments, animals and artificial agents must learn to act adaptively to maximize fitness and rewards. Such adaptive behavior can be learned through reinforcement learning1, a class of algorithms that has been successful at training artificial agents2-6 and at characterizing the firing of dopamine neurons in the midbrain7-9. In classical reinforcement learning, agents discount future rewards exponentially according to a single time scale, controlled by the discount factor. Here, we explore the presence of multiple timescales in biological reinforcement learning. We first show that reinforcement agents learning at a multitude of timescales possess distinct computational benefits. Next, we report that dopamine neurons in mice performing two behavioral tasks encode reward prediction error with a diversity of discount time constants. Our model explains the heterogeneity of temporal discounting in both cue-evoked transient responses and slower timescale fluctuations known as dopamine ramps. Crucially, the measured discount factor of individual neurons is correlated across the two tasks suggesting that it is a cell-specific property. Together, our results provide a new paradigm to understand functional heterogeneity in dopamine neurons, a mechanistic basis for the empirical observation that humans and animals use non-exponential discounts in many situations 10-14, and open new avenues for the design of more efficient reinforcement learning algorithms.

neuroscience↗

bioRxiv2017

Variability in prior expectations explains biases in confidence reports

Confidence in a decision is defined statistically as the probability of that decision being correct. Humans, however, display systematic confidence biases, as has been exposed in various experiments. Here, we show that these biases vanish when taking into account participants' prior expectations, which we measure independently of the confidence report. We use a wagering experiment to show that modeling subjects' choices allows for classifying individuals according to their prior biases, which fully explain from first principles the differences in their later confidence reports. Our parameter-free confidence model predicts two counterintuitive patterns for individuals with different prior beliefs: pessimists should report higher confidence than optimists, and, for the same task difficulty, the confidence of pessimists should increase with the generosity of the task. These findings show how systematic confidence biases can be simply understood as differences in prior expectations.

animal behavior and cognition↗