A convergent cortical scheme for visual scene segmentation

Abstract

Scene segmentation, a key step towards structured understanding of natural environments, entails a complementary set of computations. Instead of accessing these segmentation computations in isolation, we perceive the segregated object as a unified whole, but the neural basis supporting such a unitary percept remains unclear. We bridged this gap by showing a convergent coding scheme: an object-level additive code in the primary visual cortex, which serves as a spatial segmentation mask for the object. Across individual neurons, this code uniformly boosted their responses to the segmented object. In the population space, this code created an orthogonal representational geometry by perpendicularly displacing the neural manifold for image features, efficiently multiplexing both local and global information about the object. Computational modeling further revealed the utility and optimality of this convergent coding scheme, which effectively isolates objects in cluttered environments and in turn unclutters downstream object representations. Our findings reveal the computational logic of visual segmentation and call for a revision of the visual cortical hierarchy.