Manifold geometry underlies a unified code for category and category-independent features

Abstract

A central question in neuroscience and machine learning is how a single neural representation can support linear access to multiple kinds of information about the same stimulus. While this ability is widespread in biological and artificial systems, the representational principles that make such joint coding possible remain poorly understood. We address this question for an important class of representations: those jointly encoding discrete stimulus categories and continuous features that vary independently of category. Using the framework of category manifolds (the sets of neural representations elicited by stimuli from the same category) we extend existing theories of manifold classification to the equally essential task of regressing category-independent features, determining which aspects of manifold geometry govern regression performance. This provides a unified framework for understanding how classification- and regression-relevant geometry can be jointly optimized to implement an effective joint code. Applying this framework to convolutional neural networks (CNNs), we find that regression-relevant geometry can be optimized through subtle changes that largely preserve classification-relevant geometry. This explains why common representational-similarity measures previously failed to distinguish joint codes from codes optimized exclusively for classification. Motivated by prior work in visual neuroscience suggesting that macaque inferotemporal cortex may jointly encode object category and category-independent features such as object position and size, we use our framework to identify principled geometric signatures that distinguish joint codes from classification-only codes in CNNs and can be tested in future neural recordings. Finally, we characterize how these signatures are affected by common experimental constraints: limited stimulus categories and neural-population subsampling.