A Population Vector Model of Visual Working Memory for Real-World Scenes

Abstract

Visual working memory is essential for navigating through and interacting with complex real-world environments. It is therefore important to understand how natural visual inputs--characterized by complex contours, continuously varying feature gradients, and spatial relationships--are represented in working memory. However, most research in this field has focused on simplified arrays of discrete artificial objects, favoring experimental control and modeling simplicity over ecological validity. This has led to quantitative models of working memory that require inputs consisting of easily parsed objects defined by a single value along one or more simple feature dimensions. It is not clear how these models could be updated to represent complex, photograph-like scenes. To overcome this limitation, we introduce a population vector model of working memory that was designed specifically for real-world scenes. This model represents a scene as a noisy vector of neural firing rates across one or more areas of the ventral pathway, as estimated by a deep neural network model. We show that this model can account for both variations in behavioral performance and patterns of brain activity in tasks that require storing naturalistic scenes in working memory. These results demonstrate the viability of our general modeling approach, setting the stage for more sophisticated models that can fully account for the storage of real-world scenes in working memory. Public Significance StatementPeople unconsciously store visual information briefly in memory thousands of times each day and use this information to help them perform a broad range of natural tasks. Although the visual working memory system used for this purpose has been intensively studied using simple and highly controlled experimental stimuli (e.g., arrays of colored squares), there has been little progress in developing formal quantitative accounts of how real-world scenes are stored in this system. Here, we provide a new model of visual working memory that was designed for real-world scenes and can predict both behavior and brain activity when people store these scenes in memory.