The zebra finch is a social songbird that lives in large groups and produces vocal communicationcalls to facilitate social interactions. A subset of these calls can be used for individual recognition, via distinct acoustic features that are stereotyped within a bird but individualized across birds. Given their large natural group sizes and the ethological importance of individual recognition, one might expect that zebra finches would have the capacity to recognize the calls of a large number of conspecifics. In this thesis, I describe a set of neuroethological experiments to test the memory of zebra finches for individual conspecifics by their vocalizations. We hypothesized that the caudal nidopallium (NCM), a higher-order auditory region of the avian brain analogous to mammalian auditory association cortex, is involved in the learning and retention of these auditory memories. Using an operant task in which birds were trained to associate the calls of some individuals with food reward, zebra finches were found to have a large capacity for recognizing individuals by their calls and song, and that those associations could be learned with just a few training examples and persist for at least a month without reinforcement. Furthermore, lesions to NCM eliminated previously formed associations but did not prevent re-learning or learning of novel stimuli, in contrast with lesions to vocal pre-motor pathways which had no effect on the recognition ability of the birds. Finally, using the spiking activity from single neurons across the cortical-like auditory regions of the brain, we found that familiar and task-relevant vocalizations elicited more reliable neural responses, with higher information capacity, than in response to unfamiliar and less behaviorally relevant calls.

Mapping the functional connectome underlying recorded time-series of brain activity can reveal meaningful pathways of large-scale neural computation. However, it remains commonplace to estimate functional connectomes using pairwise correlative methods which are prone to spurious correlations and vulnerable to structured noise. Multivariate statistical estimates of functional connectivity offer new possibilities for generating parsimonious networks. We generated functional connectomes from resting state EEG (electroencephalography) and fMRI (functional magnetic resonance imaging) collected from infants and children under the age of 3. We generated directed multivariate functional connectomes using the ensemble statistical framework Union of Intersections (UoI) to perform regularized multivariate linear regression. We also generated pairwise connections using the Pearson correlation coefficient between each pair of time series in the recording. We found that multivariate estimates of functional connectivity from EEG are sparse and small-world, while pairwise connectomes are lattice-like and spatially correlated. We found a significant difference in small-worldness $\omega$ ($p<<0.0001$) and proximity dependence of coupling strength ($p<< 0.0001$) between pairwise and multivariate EEG functional connectomes. Functional connectomes generated from fMRI were spatially distributed for both pairwise and multivariate estimates, but we observed a significant difference in $\omega$ ($p<<0.001$), with multivariate connectomes clustered tightly around 0, indicating small-worldness. We observed lateralized structure in the multivariate fMRI connectomes that remained stable across age groups. In particular, coupling between the language network posterior superior temporal gyrus (pSTG) and salience network supramarginal gyrus (SMG) showed strong positive ipsilateral connections and strong negative contralateral connections, reinforcing the known lateralization of the language network. The same grid-like structure was observed in sensorimotor lateral fields coupling with the dorsal attention network intraparietal sulcus (IPS). This lateralized structure was not found in pairwise estimates.

We used fMRI functional connectomes to predict cognitive development scores assessed with the Mullen Scales of Early Learning (MSEL). We partitioned subjects into three groups using unsupervised clustering based on their raw MSEL scores. Multiple feature sets were extracted from the functional connectomes, by principal components analysis (PCA) and sparse PCA, and by selecting task-relevant functional network pairs. Random forest classifiers were used to predict the Mullen groups from neural feature vectors. We found that sparse PCs predicted raw scores significantly better than chance for multivariate connectomes, while both PCs and sparse PCs scored significantly better than chance for pairwise connectomes. For multivariate connectomes, functional connectivity between language and salience (SN) scored better than chance. For pairwise connectomes, coupling between the default mode network (DMN) and the frontoparietal network (FPN), and between SN and DMN scored better than chance.

Our understanding of distributed representations of complex sounds in auditory cortex remains incomplete. This is in part due a lack of experimental data for neural responses to complex naturalistic stimuli. Here, we describe the development of a large, diverse dataset of natural sounds. The sources and semantic content of the sounds are described, and acoustic features are calculated for a hand-selected set of 99 high-quality sounds and used to predict semantic labels using a support vector machine. We briefly discuss the intended use case of the sound database as a stimulus set for the characterization of neural representations of natural sound statistics, and selection for features that discriminate semantic categories.

In progress

We undertook a detailed analysis of population spike rate and LFP power in the Zebra finch auditory system. Utilizing the full range of Zebra finch vocalizations and dual-hemisphere multielectrode recordings from auditory neurons, we used encoder models to show how intuitive acoustic features such as amplitude, spectral shape and pitch drive the spike rate of individual neurons and LFP power on electrodes. Using ensemble decoding approaches, we show that these acoustic features can be successfully decoded from the population spike rate vector and the power spectra of the multielectrode LFP with comparable performance. In addition we found that adding pairwise spike synchrony to the spike rate decoder boosts performance above that of the population spike rate alone, or LFP power spectra. We also found that decoder performance grows quickly with the addition of more neurons, but there is notable redundancy in the population code. Finally, we demonstrate that LFP power on an electrode can be well predicted by population spike rate and spike synchrony. High frequency LFP power (80-190Hz) integrates neural activity spatially over a distance of up to 250 microns, while low frequency LFP power (0-30Hz) can integrate neural activity originating up to 800 microns away from the recording electrode.

To understand how an auditory system processes complex sounds, it is essential to understand how the temporal envelope of sounds, i.e. the time-varying amplitude, is encoded by neural activity. We studied the temporal envelope of Zebra finch vocalizations, and show that it exhibits modulations in the 0-30Hz range, similar to human speech. We then built linear filter models to predict 0-30Hz LFP activity from the temporal envelopes of vocalizations, achieving surprisingly high performance for electrodes near thalamorecipient areas of Zebra finch auditory cortex. We then show that there are two spatially-distinct subnetworks that resonate at different frequency bands, one subnetwork that resonates around 19Hz, and another subnetwork that resonates at 14Hz. These two subnetworks are present in every anatomical region. Finally we show that we can improve predictive performance with recurrent neural network models.

Natural sounds include animal vocalizations, environmental sounds such as wind, water and fire noises and non-vocal sounds made by animals and humans for communication. These natural sounds have characteristic statistical properties that make them perceptually salient and that drive auditory neurons in optimal regimes for information transmission.

Recent advances in statistics and computer sciences have allowed neuro-physiologists to extract the stimulus-response function of complex auditory neurons from responses to natural sounds. These studies have shown a hierarchical processing that leads to the neural detection of progressively more complex natural sound features and have demonstrated the importance of the acoustical and behavioral contexts for the neural responses.

High-level auditory neurons have shown to be exquisitely selective for conspecific calls. This fine selectivity could play an important role for species recognition, for vocal learning in songbirds and, in the case of the bats, for the processing of the sounds used in echolocation. Research that investigates how communication sounds are categorized into behaviorally meaningful groups (e.g. call types in animals, words in human speech) remains in its infancy.

Animals and humans also excel at separating communication sounds from each other and from background noise. Neurons that detect communication calls in noise have been found but the neural computations involved in sound source separation and natural auditory scene analysis remain overall poorly understood. Thus, future auditory research will have to focus not only on how natural sounds are processed by the auditory system but also on the computations that allow for this processing to occur in natural listening situations.

The complexity of the computations needed in the natural hearing task might require a high-dimensional representation provided by ensemble of neurons and the use of natural sounds might be the best solution for understanding the ensemble neural code.

This dissertation investigates the cortical representation of speech perception, using a combination of functional Magnetic Resonance Imaging (fMRI) and psychoacoustical experiments.

Previous research has shown that low-level acoustical structure, phonemes, and words are processed by distinct cortical areas. However, little is known about the relationship between these different representations. To address this problem we simultaneously mapped many different representations of speech. We recorded fMRI responses from subjects listening to over two hours of natural speech. We then examined three features spaces representing the speech sounds in terms of auditory, articulatory and semantic features. We used voxel-wise modeling for each feature space combined with a novel variance-partitioning method to assess how much response variance could be explained uniquely by each model or jointly between two or three models. Validating our approach, we found that a quarter of the brain was significantly responsive to the stories, and that our models could account for up to 45% of the explainable variance in cortex and over 60% of the explainable variance in auditory areas. We also found a hierarchical set of processing steps starting in primary auditory areas and moving along the posteroventral region of the temporal lobe that are involved in the sound to word meaning transformation.

The second part of this dissertation is a psychoacoustical investigation of the modulation power spectrum (MPS) of speech. The MPS is obtained by taking the 2-dimensional Fourier transform of the speech spectrogram. We showed that comprehension of vowels and consonants is differently affected by removal of specific spectral or temporal modulations. Supplementary consonant analysis showed differences in MPS and psychoacoustical comprehension results between three groups of consonants, separated based on the manner in which they are pronounced (fricatives, stops, and sonorants). The MPS could serve as an excellent intermediate step between lower and higher levels of speech processing, and could in future studies add nuance to our previous three cortical models of speech perception.

Although a universal code for the acoustic features of animal vocal communication calls may not exist, the thorough analysis of the distinctive acoustical features of vocalization categories is important not only to decipher the acoustical code for a specific species but also to understand the evolution of communication signals and the mechanisms used to produce and understand them.

Here, we recorded more than 8,000 examples of almost all the vocalizations of the domesticated Zebra finch, Taeniopygia guttata: vocalizations produced to establish contact, to form and maintain pair bonds, to sound an alarm, to communicate distress or to advertise hunger or aggressive intents. We characterized each vocalization type using complete representations that avoided any a priori assumptions on the acoustic code, as well as classical bioacoustics measures that could provide more intuitive interpretations. We then used these acoustical features to rigorously determine the potential information-bearing acoustical features for each vocalization type using both a novel regularized classifier and an unsupervised clustering algorithm. Vocalization categories are discriminated by the shape of their frequency spectrum and by their pitch saliency (noisy to tonal vocalizations) but not particularly by their fundamental frequency. Notably, the spectral shape of zebra finch vocalizations contains peaks or formants that vary systematically across categories and that would be generated by active control of both the vocal organ (source) and the upper vocal tract (filter).

Animals and humans are able to communicate vocally in very challenging acoustic conditions. Background noise, especially from other individuals of the same species, may mask the relevant signal and propagation can introduce significant distortions to the sound waveform. While our brains are able to extract meaningful information from heavily degraded communication sounds, the mechanisms by which the auditory system performs this task are not well understood. This thesis shows how neural systems can and do handle signal degradations. by examining how auditory neurons in an animal model of communication, the Zebra Finch Taeniopygia guttata, process degraded and undegraded signals, and demonstrates that these principles can be used to perform noise reduction on voice recordings. I discuss how the notion of invariance, common in studies of sensory perception for roughly a century, has more recently been helpful in the analysis of sensory systems at the neural level. I discuss how to characterize the invariant properties of vocal sounds, and how to connect this analysis to the mathematical theory of invariants.

To characterize invariance at the neural level, I construct a novel metric using spike-train cross-correlation between neural responses to the same signal obtained under various conditions. Using this measure, I show that a subset of neurons in avian secondary auditory forebrain area NCM can extract a representation of birdsong that is robust to background noise interference. Spectro-temporal receptive field (STRF) and modulation transfer function (MTF) analysis show that these invariant neurons are sensitive to slowly changing pitch features. Then, using stimuli that have been degraded systematically along spectral and temporal features, I further characterize the nature and origin of invariant response properties in neurons throughout avian auditory forebrain. The response of auditory neurons to spectral degradation is well explained by their MTF, but results in the temporal domain show that some neurons show invariance properties beyond those expected from this model.

Finally, I use the insights from these experiments to construct a noise-reduction algorithm that can be implemented in real-time on digital systems. The system performs well when compared to state-of-the-art algorithms for noise reduction and I discuss how these systems interrelate in terms of processing the statistics of vocal sounds. Using these comparisons and interpretations, I show how we might improve the performance of such noise reduction algorithms.