UCSF

Recent research has explored the functional role of the human auditory and sensorimotor cortices in perceiving and producing speech. One key finding from these studies was the characterization of how phonetic features and phonemes, which are fundamental units of speech, are represented in the brain. In this thesis, I examine how these neural representations of speech can be used as the basis for decoding what individuals hear and say in real-time. This work leverages recent advances in human neurophysiology with epilepsy patients that have high-density electrocorticography arrays implanted on the surface of their brains as part of their clinical treatment, enabling collection of cortical activity at unprecedented spatiotemporal resolutions. Using these signals, I show that spatiotemporal feature vectors containing cortical activity in the high gamma frequency band can be used to decode speech sounds that listeners perceive by employing techniques from automatic speech recognition, contributing to an emerging field of research referred to as neural speech recognition (NSR). Next, I design and evaluate a real-time system capable of reliably classifying aurally presented sentences using phoneme-level models and spatiotemporal high gamma features. Finally, I demonstrate state-of-the-art real-time decoding of perceived and produced words and sentences in a naturalistic question-and-answer paradigm, illustrating the utility of NSR in a real-world interactive application. In addition to characterizing properties of speech that can be decoded from the human brain in real-time, these findings have practical implications for the design of speech neuroprostheses to aid patients who are unable to communicate due to paralysis.