Brain-wide distributed processing underlying natural vision and audition

Abstract

A key challenge in neuroscience is understanding how large-scale brain dynamics give rise to perception under naturalistic conditions. Here we address this question using multivariate information decomposition of functional interactions between brain regions while typically developed, congenitally blind and congenitally deaf participants experienced audiovisual, auditory-only or visual-only versions of the same narrative. This approach allowed us to resolve information dynamics into redundant components, indexing shared information and sensory robustness, and synergistic components, indexing distributed processing that emerges only through joint interactions between brain regions. Redundant interactions were closely aligned with conventional functional connectivity and captured sensory modality- and experience-dependent differences mainly within sensory systems, whereas synergistic interactions captured these differences across high-level cortical areas. Crucially, prefrontal cortex coordinated brain dynamics through synergistic interactions independently of sensory modality and previous sensory experience, consistent with a modality-independent hub that encodes high-level semantic information and orchestrates the demands of naturalistic perception. Moreover, multimodal perception elicited the strongest high-order synergistic interactions across distributed brain subsystems, followed by unimodal visual and auditory perception, whereas sensory deprivation reduced this distributed architecture. Finally, synergistic and redundant interactions jointly supported the encoding of low-level sensory and high-level perceptual features, revealing how the trade-off between sensory robustness and distributed processing is organized across the human brain. Together, these results reveal a brain-wide information architecture for natural vision and audition, illuminating how distributed processing in the human brain supports rich perceptual experience.