Multisensory Continuous Psychophysics: Perceived Visual Object Location is Improved by Auditory Cues
Continuous Psychophysics, which couples a continuous stimulus with a continuous response, is a promising tool to break out of the confines of traditional designs based on discrete trials. In this pre-registered study, we explore to what extent this paradigm is useful in the study of multisensory integration. We expand on Tonelli et al.s (2025) seminal study by additionally examining the role of eye-movements, using a Kalman filter to estimate the sensory noise underlying behavioral tracking parameters and employing a virtual reality set-up. We immersed two cohorts of participants (n = 30 each) in a virtual meadow environment and asked them to continuously track a drone (Experiment 1) or a swarm of flies (Experiment 2) with a controller, while simultaneously recording their eye movements. We manipulated the reliability of visual cues using four levels of fog (from a completely clear view to impenetrable fog where no visual cues to the targets position were available) as well as the presence of sound cues emitted from the object (sound present/absent). The maximum correlation between stimulus and response was higher when sound was present in some conditions, particularly when visual uncertainty was high, while the tracking delay remained unaffected across all fog levels. Using a Kalman filter to estimate the underlying sensory noise, we found strong evidence that sensory noise was lower when sound was present than when sound was absent both for manual and for ocular tracking, particularly for those conditions with higher visual uncertainty. In exploratory analyses, we further show strong correlations between manual and ocular tracking in all measures (maximum correlation, tracking delay, sensory precision). However, when isolating the multisensory advantage, these correlations all but disappeared for maximum correlation and tracking delay, while remaining substantial for sensory precision. Similarly, behavioral tracking correlated generally strongly with underlying sensory noise, but much less so when it came to the advantage conferred by added sound cues. Our results show that continuous psychophysics is well-suited for the study of multisensory integration, particularly when a Kalman filter analysis is used to estimate sensory uncertainty from behavioral data.