Correctness is its own reward: bootstrapping error signals in self-guided reinforcement learning

Abstract

Reinforcement learning (RL) offers a compelling account of how agents learn complex behaviors by trial and error, yet RL is predicated on the existence of a reward function provided by the agents environment. By contrast, many skills are learned without external guidance, posing a challenge to RLs ability to account for self-directed learning. For instance, juvenile male zebra finches first memorize and then train themselves to reproduce the song of an adult male tutor through extensive practice. This process is believed to be guided by an internally computed assessment of performance quality, though the mechanism and development of this signal remain unknown. Here, we propose that, contrary to prevailing assumptions, tutor song memorization and performance assessment are subserved by the same neural circuit, one trained to predictively cancel tutor song. To test this hypothesis, we built models of a local forebrain circuit that uses contextual premotor signals to cancel tutor song auditory input via synaptic plasticity. After learning, excitatory projection neurons signaled mismatches between the tutor song and birds own performance, best matching experimental data when learning involved anti-Hebbian plasticity in recurrent interneurons. We also found that learning proceeds by both sharpening error sensitivity and minimizing responses to the tutor song. Finally, the error signals produced by this model can train a simple RL agent to replicate the spectrograms of adult bird songs. These results suggest that local learning via predictive cancellation suffices for bootstrapping error signals capable of guiding self-directed learning of natural behaviors. SignificanceMany species, including humans, learn new skills by trial and error. But in the absence of external rewards and punishments, individuals need a means of assessing the quality of their performance. Here, we model the process by which juvenile male zebra finches memorize the song of an adult tutor, showing how local changes in brain circuits responsible for hearing can give rise during this process to an internal error signal that later allows for self-guided learning.