The effects of object category training on the responses of macaque inferior temporal cortex are consistent with performance-optimizing updates within a visual hierarchy

Abstract

How does the primate brain coordinate plasticity when learning to discriminate new objects? We measured consequences of object learning on macaque inferior temporal (IT) cortex, a key waypoint supporting object recognition in the ventral visual stream. Neural activity in task-trained monkeys IT showed increased object selectivity, enhanced linear separability across objects, and more object-invariant representations compared to task-naive monkeys. To model these differences, we developed a computational framework using anatomically-mapped artificial neural network (ANN) models of the ventral stream with various learning algorithms. Simulations revealed that gradient-based, performance-optimizing updates of ANN internal representations accurately approximated observed IT cortex changes. These models predict novel training-induced phenomena in IT cortex, including changes independent of object identity and ITs alignment with behavior. This convergence between empirical measurements and model predictions suggests ventral stream plasticity follows task optimization principles well-approximated by gradient descent, enabling accurate predictions about visual plasticity and generalization to test images.