Paradoxical replay can protect contextual task representations from destructive interference when experience is unbalanced

Abstract

Experience replay is a powerful mechanism to learn efficiently from limited experience. Despite decades of compelling experimental results, the factors that determine which experiences are selected for replay remain unclear. A particular challenge for current theories is a set of studies reporting "paradoxical" replay of trajectories that are neither experienced recently or about to be chosen. To understand why, we simulated a feedforward neural network with two regimes: rich learning (structured representations tailored to task demands) and lazy learning (unstructured, task-agnostic representations). Rich, but not lazy, representations degraded following unbalanced experience, an effect that could be reversed with replay of non-chosen options. To test if this computational principle can account for the experimental data, we examined the relationship between paradoxical replay and learned task representations in the rat hippocampus. Across two different studies, we found an association between the richness of learned task representations and the paradoxicality of replay. Taken together, these results suggest that paradoxical replay may serve to protect rich representations from the destructive effects of unbalanced experience, and more generally demonstrate a novel interaction between the nature of task representations and the function of replay in artificial and biological systems. Significance StatementWe provide an explicit normative explanation and simulations of the experimentally observed puzzle of "paradoxical" replay of experiences that are neither experienced recently or about to be chosen. We show computationally that such replay can serve to protect certain task representations from the destructive effects of unbalanced experience. We experimentally confirm the main prediction of the theory, that "rich" task representations, measured using representational distance in the rodent hippocampus, show more paradoxical replay compared to "lazy" task representations. Our theory refines the notion of consolidation in complementary learning systems theory in showing that not all task representations benefit equally from interleaving, and provides an example of how the use of replay in artificial neural networks can be optimized.