Visual Semantic Encoding and Identification of Naturalistic Movies via High-Density Diffuse Optical Tomography

Abstract

Understanding how the brain represents meaning in real-world contexts is essential for both fundamental neuroscience and clinical applications. Brain encoding and decoding models from naturalistic stimuli provide a powerful window into semantic representations. Yet, existing approaches rely on a constrained scanning environment, or on conventional fNIRS, which has been limited to sparse sampling and/or block-design paradigms. Here, we tested whether high-density diffuse optical tomography (HD-DOT), an advanced high-density tomographic optical imaging method, can support semantic encoding and decoding using naturalistic movies. We collected 3.5 hours of naturalistic movie viewing data from six participants using stimuli labeled with 1,708 categories. Encoding models robustly predicted voxel-level responses, yielding single semantic category maps consistent with prior fMRI studies. In complementary decoding analyses, we showed that DOT responses captured sufficient semantic content to identify which clips participants viewed. To assess organization across individuals, we identified a shared low-dimensional semantic space that captures common semantic dimensions. Finally, clustering analyses revealed interpretable higher-order semantic dimensions like social and animate agents, objects vs natural organisms, and textural scenes, consistently mapped across the cortex. These findings demonstrate that DOT can recover distributed, high-dimensional semantic representations from naturalistic movies, bridging fMRI-level semantic mapping with the accessibility of optical imaging.