Words in context: tracking context-processing during language comprehension using computational language models and MEG

Abstract

The meaning of a word depends on its lexical semantics and on the context in which it is embedded. At the basis of this lays the distinction between lexical retrieval and integration, two basic operations supporting language comprehension. In this paper, we investigate how lexical retrieval and integration are implemented in the brain by comparing MEG activity to word representations generated by computational language models. We test both non-contextualized embeddings, representing words independently from their context, and contextualized embeddings, which instead integrate contextual information in their representations. Using representational similarity analysis over cortical regions and over time, we observed that brain activity in the left anterior temporal pole and inferior frontal regions shows higher similarity with contextualized word embeddings compared to non-contextualized embeddings, between 300 and 500 ms after word presentation. On the other hand, non-contextualized word embeddings show higher similarity with brain activity in the left lateral and anterior temporal lobe at earlier latencies - areas and latencies related to lexical retrieval. Our results highlight how lexical retrieval and context integration can be tracked in the brain using word embeddings obtained with computational models. These results also suggest that the distinction between lexical retrieval and integration might be framed in terms of context-independent and contextualized representations.