Audiovisual adaptation is expressed in spatial and decisional codes

Abstract

The brain adapts dynamically to the changing sensory statistics of its environment. The neural circuitries and representations that support this cross-sensory plasticity remain unknown. We combined psychophysics and model-based representational fMRI and EEG to characterize how the adult human brain adapts to misaligned audiovisual signals. We show that audiovisual adaptation moulds regional BOLD-responses and fine-scale activity patterns in a widespread network from Heschls gyrus to dorsolateral prefrontal cortices. Crucially, audiovisual recalibration relies on distinct spatial and decisional codes that are expressed with opposite gradients and timecourses across the auditory processing hierarchy. Early activity patterns in auditory cortices encode sounds in a continuous space that flexibly adapts to misaligned visual inputs. Later activity patterns in frontoparietal cortices code decisional uncertainty consistent with these spatial transformations. Our findings demonstrate that regions throughout the auditory processing hierarchy multiplex spatial and decisional codes to adapt flexibly to the changing sensory statistics in the environment.