Predictive coding video models capture dorsal parietal representations and human judgments for surfaces defined by motion

Abstract

Interacting with the world relies on knowing its three-dimensional structure, yet vision receives a stream of two-dimensional images. Recovering 3D geometry from dynamic input is therefore foundational to vision, but lacks a computational account. Here we identify one, using artificial neural networks to predict human behavior and neuronal responses from the two major visual pathways in macaques. Using videos of camouflaged objects whose shape is revealed through motion, we found both pathways carry information for motion-defined surfaces, while neurons in parietal cortex most closely track what humans perceive. Only predictive coding video models--trained to fill in missing content in natural videos--reproduced both neuronal responses and human behavior. Predictive coding thus emerges as a principle that yields a brain-like geometric world model of the physical world.