Optimal motion-in-depth estimation with natural stimuli

Abstract

Estimating the motion of objects in depth is important for behavior, and is strongly supported by binocular visual cues. To understand both how the brain should estimate motion in depth and how natural constraints shape and limit performance, we develop image-computable ideal observer models from naturalistic binocular video clips of two 3D motion tasks. The observers spatio-temporally filter the videos, and non-linearly decode 3D motion from the filter responses. The optimal filters and decoder are dictated by the task-relevant natural image statistics, and are specific to each task. Multiple findings emerge. First, two distinct filter types are spontaneously learned for each task. For 3D speed estimation, filters emerge for processing either changing disparities over time (CDOT) or interocular velocity differences (IOVD), cues used by humans. For 3D direction estimation, filters emerge for discriminating either left-right or towards-away motion. Second, the filter responses, conditioned on the latent variable, are well-described as jointly Gaussian, and the covariance of the filter responses carries the information about the task-relevant latent variable. Quadratic combination is thus necessary for optimal decoding, which can be implemented by biologically plausible neural computations. Finally, the ideal observer yields non-obvious-and in some cases counter-intuitive-patterns of performance like those exhibited by humans. Important characteristics of human 3D motion processing and estimation may therefore result from optimal information processing in the early visual system. Significance statementHumans and other animals extract and process features of natural images that are useful for estimating motion-in-depth, an ability that is crucial for successful interaction with the environment. But the enormous diversity of natural visual inputs that are consistent with a given 3D motion-natural stimulus variability-presents a challenging computational problem. The neural populations that support the estimation of motion-in-depth are under active investigation. Here, we study how to optimally estimate local 3D motion with naturalistic stimulus variability. We show that the optimal computations are biologically plausible, and that they reproduce sometimes counterintuitive performance patterns independently reported in the human psychophysical literature. Novel testable hypotheses for future neurophysiological and psychophysical research are discussed.