Across-speaker articulatory reconstruction from sensorimotor cortex for generalizable brain-computer interfaces

Abstract

ObjectiveSpeech brain-computer interfaces (BCIs) can restore speech features like articulatory movements from brain activity. However, for individuals with vocal tract paralysis, lack of articulatory movements can pose a challenge for speech BCI development. To address this challenge, our study aims at extracting generalizable articulatory features from a group of native Dutch speakers and reconstructing these features from brain data of a separate group of able-bodied individuals. ApproachWe applied a tensor component analysis (TCA) model to extract generalisable articulatory features from a publicly available articulatory movement dataset. To reconstruct articulatory features from the brain, we analyzed data of three able-bodied participants P1, P2 and P3 with high-density electrocorticography (HD-ECoG) electrode arrays implanted over the sensorimotor cortex. For each participant, a separate TCA model was applied to extract neural features. A gradient boosting regression model was used to reconstruct articulatory features from neural features. Reconstruction performance was measured as Pearsons correlation coefficient (PCC) between the reconstructed and the generalizable articulatory features. ResultsThe extracted articulatory features showed even contributions across participants, indicating that these features captured generalizable articulatory kinematic patterns. By using these features, we were able to reconstruct articulatory features from brain data. PCC between the reconstructed and original articulatory features were significantly above chance for all three participants, with mean PCCs of 0.80, 0.75 and 0.76 for P1, P2 and P3 respectively. SignificanceWith the rapid development of speech BCI, our research demonstrates that speech-related articulatory features can be restored from HD-ECoG signal using generalizable articulatory features derived from able-bodied individuals. With the potential to reconstruct audio or speech-related facial movements from the reconstructed articulatory features, our framework may provide a new way for developing speech BCIs for people unable to produce mouth movements.