Aubrey, B. | Search bioRxiv

bioRxiv2020

Moment-by-moment tracking of naturalistic learning and its underlying hippocampo-cortical interactions

Every day our memory system achieves a remarkable feat: We form lasting memories of stimuli that were only encountered once. Here we investigate such learning as it naturally occurs during story listening, with the goal of uncovering when and how memories are stored and retrieved during processing of continuous, naturalistic stimuli. In behavioral experiments we confirm that, after a single exposure to a naturalistic story, participants can learn about its structure and are able to recall upcoming words in the story. In patients undergoing electrocorticographic recordings, we then track mnemonic information in high frequency activity (70 - 200Hz) as patients listen to a story twice. In auditory processing regions we demonstrate the rapid reinstatement of upcoming information after a single exposure; this neural measure of predictive recall correlates with behavioral measures of event segmentation and learning. Connectivity analyses on the neural data reveal information-flow from cortex to hippocampus at the end of events. On the second time of listening information-flow from hippocampus to cortex precedes moments of successful reinstatement.

neuroscience↗

bioRxiv2020

Thinking ahead: prediction in context as a keystone of language in humans and machines

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models are trained to generate appropriate linguistic responses in a given context. We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process natural language: 1) both are engaged in continuous next-word prediction before word-onset; 2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise (i.e., prediction error signals); 3) both represent words as a function of the previous context. In support of these three principles, our findings indicate that: a) the neural activity before word-onset contains context-dependent predictive information about forthcoming words, even hundreds of milliseconds before the words are perceived; b) the neural activity after word-onset reflects the surprise level and prediction error; and c) autoregressive DLM contextual embeddings capture the neural representation of context-specific word meaning better than arbitrary or static semantic embeddings. Together, our findings suggest that autoregressive DLMs provide a novel and biologically feasible computational framework for studying the neural basis of language.

neuroscience↗