Continuous and discrete decoding of overt speech with electroencephalography

Abstract

Neurological disorders affecting speech production adversely impact quality of life for over 7 million individuals in the US. Traditional speech interfaces like eye-tracking devices and P300 spellers are slow and unnatural for these patients. An alternative solution, speech Brain-Computer Interfaces (BCIs), directly decodes speech characteristics, offering a more natural communication mechanism. This research explores the feasibility of decoding speech features using non-invasive EEG. Nine neurologically intact participants were equipped with a 63-channel EEG system with additional sensors to eliminate eye artifacts. Participants read aloud sentences displayed on a screen selected for phonetic similarity to the English language. Deep learning models, including Convolutional Neural Networks and Recurrent Neural Networks with/without attention modules, were optimized with a focus on minimizing trainable parameters and utilizing small input window sizes. These models were employed for discrete and continuous speech decoding tasks, achieving above-chance participant-independent decoding performance for discrete classes and continuous characteristics of the produced audio signal. A frequency sub-band analysis highlighted the significance of certain frequency bands (delta, theta, and gamma) for decoding performance, and a perturbation analysis identified crucial channels. Assessed channel selection methods did not significantly improve performance, but they still outperformed chance levels, suggesting a distributed representation of speech information encoded in the EEG signals. Leave-One-Out training demonstrated the feasibility of utilizing common speech neural correlates, reducing data collection requirements from individual participants.