Biology subjects

Millidge, B.

Publications and source records attributed to Millidge, B..

3 recordsLinked to original sources

bioRxiv2023

Predictive Coding Networks for Temporal Prediction

One of the key problems the brain faces is inferring the state of the world from a sequence of dynamically changing stimuli, and it is not yet clear how the sensory system achieves this task. A well-established computational framework for describing perceptual processes in the brain is provided by the theory of predictive coding. Although the original proposals of predictive coding have discussed temporal prediction, later work developing this theory mostly focused on static stimuli, and key questions on neural implementation and computational properties of temporal predictive coding networks remain open. Here, we address these questions and present a formulation of the temporal predictive coding model that can be naturally implemented in recurrent networks, in which activity dynamics rely only on local inputs to the neurons, and learning only utilises local Hebbian plasticity. Additionally, we show that temporal predictive coding networks can approximate the performance of the Kalman filter in predicting behaviour of linear systems, and behave as a variant of a Kalman filter which does not track its own subjective posterior variance. Importantly, temporal predictive coding networks can achieve similar accuracy as the Kalman filter without performing complex mathematical operations, but just employing simple computations that can be implemented by biological networks. Moreover, when trained with natural dynamic inputs, we found that temporal predictive coding can produce Gabor-like, motion-sensitive receptive fields resembling those observed in real neurons in visual areas. In addition, we demonstrate how the model can be effectively generalized to nonlinear systems. Overall, models presented in this paper show how biologically plausible circuits can predict future stimuli and may guide research on understanding specific neural circuits in brain areas involved in temporal prediction. Author summaryWhile significant advances have been made in the neuroscience of how the brain processes static stimuli, the time dimension has often been relatively neglected. However, time is crucial since the stimuli perceived by our senses typically dynamically vary in time, and the cortex needs to make sense of these changing inputs. This paper describes a computational model of cortical networks processing temporal stimuli. This model is able to infer and track the state of the environment based on noisy inputs, and predict future sensory stimuli. By ensuring that these predictions match the incoming stimuli, the model is able to learn the structure and statistics of its temporal inputs and produces responses of neurons resembling those in the brain. The model may help in further understanding neural circuits in sensory cortical areas.

neuroscience↗

bioRxiv2022

Recurrent predictive coding models for associative memory employing covariance learning

The computational principles adopted by the hippocampus in associative memory (AM) tasks have been one of the mostly studied topics in computational and theoretical neuroscience. Classical models of the hippocampal network assume that AM is performed via a form of covariance learning, where associations between memorized items are represented by entries in the learned covariance matrix encoded in the recurrent connections in the hippocampal subfield CA3. On the other hand, it has been recently proposed that AM in the hippocampus is achieved through predictive coding. Hierarchical predictive coding models following this theory perform AM, but fail to capture the recurrent hippocampal structure that encodes the covariance in the classical models. Such a dichotomy pose potential difficulties for developing a unitary theory of how memory is formed and recalled in the hippocampus. Earlier predictive coding models that learn the covariance information of inputs explicitly seem to be a solution to this dichotomy. Here, we show that although these models can perform AM, they do it in an implausible and numerically unstable way. Instead, we propose alternatives to these earlier covariance-learning predictive coding networks, which learn the covariance information implicitly and plausibly, and can use dendritic structures to encode prediction errors. We show analytically that our proposed models are perfectly equivalent to the earlier predictive coding model learning covariance explicitly, and encounter no numerical issues when performing AM tasks in practice. We further show that our models can be combined with hierarchical predictive coding networks to model the hippocampo-neocortical interactions. Our models provide a biologically plausible approach to modelling the hippocampal network, pointing to a potential computational mechanism employed by the hippocampus during memory formation and recall, which unifies predictive coding and covariance learning based on the recurrent network structure. Author summaryThe hippocampus and adjacent cortical areas have long been considered essential for the formation of associative memories. Earlier theoretical works have assumed that the hippocampus stores in its recurrent connections statistical regularities embedded in the sensory inputs. On the other hand, it has been recently suggested that the hippocampus retrieves memory by generating predictions of ongoing sensory inputs. Computational models have thus been proposed to account for this predictive nature of the hippocampal network using predictive coding, a general theory of information processing in the cortex. However, these hierarchical predictive coding models of the hippocampus did not describe how it stores the statistical regularities that play a key role for associative memory in the classical hippocampal models, hindering a unified understanding of the underlying computational principles employed by the hippocampus. To address this dichotomy, here we present a family of predictive coding models that also learn the statistical information needed for associative memory. Our models can stably perform associative memory tasks in a biologically plausible manner, even with large structured data such as natural scenes. Our work provides a possible mechanism of how the recurrent hippocampal network may employ various computational principles concurrently to perform associative memory.

neuroscience↗

bioRxiv2022

Reward Bases: Instantaneous reward revaluation with temporal difference learning

AO_SCPLOWBSTRACTC_SCPLOWAn influential theory posits that dopaminergic neurons in the mid-brain implement a model-free reinforcement learning algorithm based on temporal difference (TD) learning. A fundamental assumption of this model is that the reward function being optimized is fixed. However, for biological creatures the reward function can fluctuate substantially over time depending on the internal physiological state of the animal. For instance, food is rewarding when you are hungry, but not when you are satiated. While a variety of experiments have demonstrated that animals can instantly adapt their behaviour when their internal physiological state changes, under current thinking this requires model-based planning since the standard model of TD learning requires retraining from scratch if the reward function changes. Here, we propose a novel and simple extension to TD learning that allows for the zero-shot (instantaneous) generalization to changing reward functions. Mathematically, we show that if we assume the reward function is a linear combination of reward basis vectors, and if we learn a value function for each reward basis using TD learning, then we can recover the true value function by a linear combination of these value function bases. This representational scheme allows instant and perfect generalization to any reward function in the span of the reward basis vectors as well as possesses a straightforward implementation in neural circuitry by parallelizing the standard circuitry required for TD learning. We demonstrate that our algorithm can also reproduce behavioural data on reward revaluation tasks, predict dopamine responses in the nucleus accumbens, as well as learn equally fast as successor representations while requiring much less memory.

neuroscience↗