The human brain contains a remarkable sensory system that allows us to effortlessly process speech. The processing of speech begins at our ears, where speech sounds are converted into electrical signals that propagate up to our brain. How these signals are transformed from acoustic information into meaningful speech in the human cortex is still unknown. In this dissertation, high-density human direct cortical recordings were used to systematically detail the human speech processing system and address central linguistic theories of speech perception. In Chapter 2, we examine how the superior temporal gyrus, a brain area critically implicated in speech perception, encodes phonetic features of speech. We found single site response selectivity to distinct phonetic features and distributed population encoding mediated by acoustic properties, including pitch and voice-onset-time. In Chapter 3, we examine the representation of speech sounds in human motor cortex, a region controversially hypothesized to process articulatory gestures during perception. We found evidence that motor cortex does not represent articulatory representations of perceived actions in speech, but rather, auditory vocal information. These results are consistent with linguistic feature hierarchies organized around acoustic, rather than articulatory, features. Finally, built upon the principles of human speech electrophysiology, we developed a clinical cortical mapping tool to aid in the preservation of eloquent cortex, the details of which are described in Chapter 4. All together, this work lays a foundation for understanding the human speech processing system and developing clinical applications to aid the lives of thousands with neurological disorders, including epilepsy and brain tumors.

Speech is a fundamental human behavior, and enables the fast, effortless expression of emotions, desires, and needs. Devastating conditions like paralysis and brain-stem stroke can rob individuals of the ability to speak, even though they retain intact cognitive abilities. Brain-Computer Interface (BCI) technology offers hope for such individuals by reading out these intact neural signals using a recording device, then deciphering what the person was trying to say using machine learning and artificial intelligence. Prior to beginning this thesis, many questions remained in the development of speech-BCIs. Could speech be decoded from the brain of a person who hadn't spoken for many years and was suffering from paralysis? What algorithms could do this? What recording technologies could be used? Would the brain activity look similar to healthy speakers, or would it have evolved? Could we enable someone to speak quickly and naturalistically with these devices?

With these questions in mind, we launched the BCI restoration of arm and voice (BRAVO) clinical trial. This trial explores the use of Electrocorticography (ECoG), a high-resolution neural recording modality, to read out and decode neural activity during intended speech. This thesis presents results on work I have done as part of this clinical trial, and demonstrates successful speech decoding with two incredible participants who have each been unable to speak for over 15 years.

Chapter 1 introduces a proof-of-concept speech BCI in someone who cannot speak, showcasing real-time decoding of 50 words and restoring communication at 15 words per minute. Chapter 2 expands the scope of speech BCIs, enabling speech-based spelling with NATO codewords, achieving 29 characters per minute and allowing communication with a practically unlimited vocabulary. We also show silent-speech attempts can be decoded, and that low-frequency neural activity (between .6 and 16.67 Hz) is critical to decoding. Chapter 3 introduces a state-of-the-art BCI which can decode text from neural activity during silent speech attempts at 80 words per minute. Sound units and facial gestures were also predictable, enabling auditory and visual representations of attempted speech. These advancements illustrate the potential of speech BCIs to restore communication for those who have lost their voice, and demonstrated that speech is

Taken together, this research presents progress in the development of speech BCIs and signifies a significant step towards improving the lives of individuals with communication impairments through versatile and comprehensive speech restoration systems.

Neural devices have great potential to improve the treatment of neuropsychiatric, sensorimotor, and epileptiform disorders in humans. Recently, focus has been appropriately directed toward enabling the bidirectional flow of information between these devices and the brain with the goal of delivering electrical therapy only when necessary and in an appropriate amount. Interactions between the nervous system and the device improve when the devices are able to 1) record high quality neural signals using the appropriate number of sensors over the correct area and 2) stimulate the brain with an electrical signal that causes a desired effect.

The work presented here addresses these two major issues necessary for neural device design. In the first section, the spatial resolution of the brain at its surface is investigated and quantified using recordings from human patients with implanted ECoG sensors. In the second section, a novel ECoG sensor array with high spatial resolution is described and tested in human subjects that undergo awake brain surgery. Finally, the neural changes associated with sub-threshold and super-threshold electrical stimulation are examined and described for the first time in relation to stimulation parameters and stimulation outcome.

We found that the level of spatial resolution at the cortical surface depends on the neural frequency. For the highest neural frequency studied, the high gamma band, a denser array of sensors than what is commercially available should increase the information content. Because of this observation, we worked with Lawrence Livermore National Labs to manufacture an array of electrodes for collecting neural activity from a single human gyrus, and we observed that the increased density of electrodes did indeed improve decoding accuracy. Electrical stimulation of the sensorimotor cortex induced high gamma activity when the subject consciously perceived the stimulation, suggesting that high gamma is a reliable biomarker of effective stimulation. Together, these results suggest that devices should record from the brain’s surface at a higher density than is typically used, and that high gamma can be utilized as a feedback signal indicating whether stimulation is effective.

To speak is to dynamically orchestrate the movements of the articulators (jaw, tongue, lips, and larynx), which in turn generate speech sounds. It is an amazing mental and motor feat that is controlled by the brain and is fundamental for communication. Technology that could translate brain signals into speech would be transformative for people who are unable to communicate as a result of neurological impairments. This work first investigates how articulator movements that underlie natural speech production are represented in the brain. Building upon this, this work also presents a neural decoder that can synthesize audible speech from brain signals. Data to support these results were from direct cortical recordings of the human sensorimotor cortex while participants spoke natural sentences. Neural activity at individual electrodes encoded a diversity of articulatory kinematic trajectories (AKTs), each revealing coordinated articulator movements towards specific vocal tract shapes. The neural decoder was designed to leverage the kinematic trajectories encoded in the sensorimotor cortex which enhanced performance even with limited data. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication.

Recent research has explored the functional role of the human auditory and sensorimotor cortices in perceiving and producing speech. One key finding from these studies was the characterization of how phonetic features and phonemes, which are fundamental units of speech, are represented in the brain. In this thesis, I examine how these neural representations of speech can be used as the basis for decoding what individuals hear and say in real-time. This work leverages recent advances in human neurophysiology with epilepsy patients that have high-density electrocorticography arrays implanted on the surface of their brains as part of their clinical treatment, enabling collection of cortical activity at unprecedented spatiotemporal resolutions. Using these signals, I show that spatiotemporal feature vectors containing cortical activity in the high gamma frequency band can be used to decode speech sounds that listeners perceive by employing techniques from automatic speech recognition, contributing to an emerging field of research referred to as neural speech recognition (NSR). Next, I design and evaluate a real-time system capable of reliably classifying aurally presented sentences using phoneme-level models and spatiotemporal high gamma features. Finally, I demonstrate state-of-the-art real-time decoding of perceived and produced words and sentences in a naturalistic question-and-answer paradigm, illustrating the utility of NSR in a real-world interactive application. In addition to characterizing properties of speech that can be decoded from the human brain in real-time, these findings have practical implications for the design of speech neuroprostheses to aid patients who are unable to communicate due to paralysis.

Speech is one of the most efficient and effortless ways to communicate. Producing speech requires planning speech targets, sequencing speech-motor movements, and coordinating a dynamic system of articulators to shape breath in real time, generating the sounds we perceive and interpret as needs, ideas, and emotions. Loss of this ability, through neurodegenerative disease or paralysis, is devastating and reduces self-reported quality of life. For many with this condition, the cortical signals to control their articulators still persist--however these signals cannot be communicated to their vocal tract, due to injured or diseased descending pathways leading to vocal tract paralysis. Recent advances in neural recording hardware, our understanding of how speech production is controlled in the brain (specifically in the ventral sensorimotor cortex), and machine learning have enabled the development of speech brain computer interfaces (BCI). A direct speech BCI would translate neural signals into intended speech, restoring the ability to communicate to these individuals. This body of work first demonstrates a proof of concept that a direct speech BCI consisting of a 50-word vocabulary can be developed using high-density neural recording hardware, called electrocorticography, in a participant who cannot speak due to severe paralysis. We then built upon this proof-of-concept by developing a spelling-based speech BCI that could be controlled by silently attempted speech. The methods we used for these studies were based on our understanding of how articulatory movements are controlled by neural signals. However, much less is known about how the brain controls the upstream processes of planning and sequencing these movements. Motivated by the success of translating neuroscientific findings into BCI development, we next sought to understand how speech is sequenced in the brain. Using a task where healthy speakers spoke syllable sequences of varying complexity, we found both neural activity specific to production and widespread sustained activity associated with planning syllable sequences. This network, consisting both of areas classically considered involved in speech planning, such as Broca’s area, as well as more novel regions like the middle precentral gyrus (mPrCG), was modulated by the complexity of the sequences. However, only the mPrCG demonstrated robust sustained activity, sequence complexity encoding, and was correlated with the participants reaction time, suggesting that this area’s role in speech planning is specific to speech-motor sequencing. We confirmed this by using direct cortical stimulation, which induced speech errors only during complex sequences, in the absence of direct motor or perceptual effects. This work establishes the mPrCG as a critical node of speech-motor sequencing, redefining traditional notions of how the brain sequences and produces speech. Together, these studies demonstrate the potential of speech brain computer interfaces for restoring speech in paralyzed individuals and puts forth new possibilities for neurobiologically informed algorithms for decoding speech.