Bernstein, L. E. | Search bioRxiv

bioRxiv2021

Neural basis of learning to perceive speech through touch using an acoustic-to-vibrotactile speech sensory substitution

The goal of sensory substitution is to convey the information transduced by one sensory system through a novel sensory modality. One example is vibrotactile (VT) speech, for which acoustic speech is transformed into vibrotactile patterns. Despite an almost century-long history of studying vibrotactile speech, there has been no study of the neural bases of VT speech learning. We here trained hearing adult participants to recognize VT speech syllables. Using fMRI, we showed that both somatosensory (left post-central gyrus) and auditory (right temporal lobe) regions acquire selectivity for VT speech stimuli following training. The right planum temporale in particular was selective for both VT and auditory speech. EEG source-estimated activity revealed temporal dynamics consistent with direct, low-latency engagement of right temporal lobe following activation of the left post-central gyrus. Our results suggest that VT speech learning achieves integration with the auditory speech system by "piggybacking" onto corresponding auditory speech representations. Significance statementIn sensory substitution, the information conveyed by one sensory system is used to replace the function of another. Blind individuals, for example, can learn to use a visual-to-acoustic sensory substitution device to navigate the world. We tested the hypothesis that sensory substitution can more generally occur in typical individuals, by exploiting the existence of multi-sensory convergence areas in the brain. We trained hearing participants to recognize speech syllables presented as vibrotactile stimulation patterns. Using fMRI and EEG, we show that vibrotactile speech learning integrates with the auditory speech system by "piggybacking" onto corresponding auditory speech representations. These results suggest that the human brain can bootstrap new senses by leveraging one sensory system to process information normally processed by another.

neuroscience↗

bioRxiv2021

Metamodal Coupling of Vibrotactile and Auditory Speech Processing Systems Through Matched Stimulus Representations

It has been postulated that the brain is organized by "metamodal", sensory-independent cortical modules implementing particular computations, leading to the intriguing hypothesis that brain areas can perform tasks (such as word recognition) not just in "standard" sensory modalities but also in novel sensory modalities. Yet, evidence for this theory, especially in neurotypical subjects, has been variable. We hypothesized that effective metamodal engagement of a brain area requires congruence between the novel and standard sensory modalities not only at the task level (e.g., "word recognition") but critically also a match at the algorithmic level (in Marrs terminology), i.e., at the level of neural representation of the information of interest. To test this hypothesis, we trained participants to recognize vibrotactile versions of auditory words using two encoding schemes. The vocoded approach preserved the dynamics and representational similarities of auditory speech while the token-based approach used an abstract phoneme-based code. Although both groups learned the vibrotactile word recognition task, only in the vocoded group did trained vibrotactile stimuli recruit the auditory speech network and lead to increased coupling between somatosensory and auditory speech areas. In contrast, the token-based encoding appeared to rely on paired-associate learning. Thus, matching neural input representations is a critical factor for assessing and leveraging the metamodal potential of cortical modules.

neuroscience↗