Animals function in a 3D world in which survival depends on robust, well-controlled actions. Historically, researchers in Artificial Intelligence (AI) and neuroscience have explored sensory and motor systems independently. There is a growing body of literature in AI and neuroscience to suggest that they actually work in tandem. While there has been a great deal of work on vision and audition as sensory modalities in these fields, one could argue that a more fundamental modality in biology is haptics, or the sense of touch. In this thesis, we will look at building computational models that integrate tactile sensing with other sensory modalities to perform manipulation-like tasks in robots and discrimination tasks in mice. We will also explore the problem of inference through the lens of Markov Chain Monte Carlo methods (MCMC). We elaborate on the ideas discussed in this thesis in the introduction presented in Chapter 1.

A challenging problem one often faces when applying probabilistic mathematical models to the study of sensory-motor systems and other problems involving learning of inference is sampling. Hamiltonian Markov Chain Monte Carlo (HMC) algorithms can efficiently draw representative samples from complex probabilistic models. Most MCMC methods rely on detailed balance to ensure that we can sample from the correct distribution. This constraint can be relaxed in discrete state spaces such as those employed by HMC type methods. In Chapter 2, we study HMC methods without detailed balance to explore faster convergence. Markov jump processes are stochastic processes on discrete state space but continuous in time. In Chapter 3, we use Markov Jump Processes to simulate waiting times along with generalized detailed balance. This waiting time ,we show, helps generate samples faster. Most MCMC methods are plagued by slow simulation times on discrete computing systems. In Chapter 4, we explore HMC in analog circuits where the problem of generating samples from a distribution is mapped to the problem of sampling charge in a capacitor.

The second half of this dissertation focuses on the role of haptics in perception and action. Manipulation is a fundamental problem for artificial and biological agents. High dimensional actuators (say, fingers, trunks,etc) are really hard to control. In Chapter 5, we present an approach to learn to actuate dexterous manipulators to grasp objects in simulation. Haptics as a sensory modality is critical to many manipulation tasks. Employing haptics in high dimensional dextrous actuators is challenging. In Chapter 6, we explore how intrinsic curiosity and haptics can be used to learn exploration strategies for discrimination of objects with dextrous hands. A key component to make tactile sensing a possibility is the availability of cheap, efficient, scalable hardware. Chapter 7 presents results for tactile servoing using a physical gelsight sensor. Traditional neuroscience texts delineate sensory and motor systems as two independent systems yet recent results suggest that this may not be entirely complete. That is, there is evidence to suggest that the representations in the cortex is more distributed than is accepted. Finally in Chapter 8, we explore building a computational model of spiking neural data collected from both the barrel and motor cortices during free and active whisking. These works help towards understanding sensorimotor representations in the context of haptics and high dimensional controls. We conclude with a discussion on future directions in Chapter 9.

Despite remarkable progress in understanding the neurophysiology of cortex, fundamental questions regarding its function remain unanswered. The present study explores how a local population of neurons along a cortical column in visual cortex responds to dynamic natural stimuli. I show how a novel application of sparse coding to neural recordings obtained with high-density laminar probes can separate the complex statistical structure of this data into biophysically interpretable, underlying causes. The resulting data representation is instrumental in characterizing both spiking and local field potential activity across lamina in response to natural movies. Finally, I present a framework for understanding laminar population activity in terms of a statistical model that accounts for network interactions as well as the driving influence of the stimulus.

Learning is one of the hallmarks of human intelligence. It marks a level of flexibility and adaptation to new information that no artificial model has achieved at this point. This remarkable ability to learn makes it possible to accomplish a multitude of cognitive tasks without requiring a multitude of information from any single task. In this thesis, we describe emergent phenomena which occur during learning for artificial neural network models. First, we observe how learning well-defined tasks can lead to the emergence of structured representations. This emergent structure appears at multiple levels within these models. From semantic factors of variation appearing in the hidden units of an autoencoder to physical structure appearing at the sensory input of an attention model, learning appears to influence all parts of a model. With this in mind, we develop a new method to guide this learning process for acquiring multiple tasks within a single model. Such methods will endow neural networks with greater flexibility to adapt to new environments without sacrificing the emergent structures which have been acquired previously from learning.

My work seeks to contribute to three broad goals: predicting the computational representations found in the brain, developing algorithms that help us infer the computations that the brain performs, and producing better statistical models of natural signals. At first glance these goals may not seem compatible; however, my work finds a common thread among them through the probabilistic modeling of phase variables. My thesis is broken down into three major chapters that reflect these three goals. Within each chapter I develop novel probabilistic models of phase variables and apply these models to the invariant representation of visual motion, to the inference of connectivity in networks of coupled neural oscillators, and to the development of statistical models of edge structure in images.

First, I develop a hierarchical model of visual processing that learns from the natural world the higher-order structure of visual motion by modeling phase transformations. The model exhibits an important invariance: the model represents the way the world moves irrespective of the way it looks. This model has implications for our interpretation of biological visual processing and provides a functional roll for feedback in cortex.

Second, I present a model and estimation technique that captures the dynamics of coupled oscillator systems and recovers the interactions of the oscillators from measurements. From a statistical perspective, the model is the multivariate phase distribution analogue to the multivariate Gaussian distribution and the estimation technique is then analogous to finding the inverse covariance matrix for a Gaussian distribution. From a dynamical systems perspective, the technique provides a solution to the inverse problem of the generalized Kuramoto model and infers from measurements the true connectivity between oscillators even when phase correlations or other phase measurements would lead to false conclusions. This technique can be broadly applied to a range of neurobiological phenomena including the inference of cortical dynamic functional networks from phase measurements.

Third, I present a model that captures aspects of the local phase structure of edges in images. We first explore the pairwise phase statistics of local, oriented filters in response to natural images and determine that pairwise phase relationships do not explain the `interesting' relationships in natural images, such as long range phase alignments. Given this finding we develop a conditional latent variable model that captures the non-stationary phase structure produced by continuous edges. This model is capable of generating long range, continuous edge structure, a hallmark of natural images.

The major contributions of this thesis can be divided into two types. First, this work provides demonstrative examples of how multivariate phase distributions may be modeled in a probabilistic framework. My hope is that the models I have developed will provide the basis for additional exploration of the mathematical development of probabilistic models of phase. Second, the results obtained from applying these models have important implications for understanding invariant visual representations of motion, investigating coherence mediated intracortical communication, and describing the statistical structure of edges in natural images.

This dissertation presents three contributions on unsupervised learning. First, I describe a signal representation framework called the sparse manifold transform that combines key ideas from sparse coding, manifold learning, and slow feature analysis. It turns non-linear transformations in the primary sensory signal space into linear interpolations in a representational embedding space while maintaining approximate invertibility. The sparse manifold transform is an unsupervised and generative framework that explicitly and simultaneously models the sparse discreteness and low-dimensional manifold structure found in natural scenes. When stacked, it also models hierarchical composition. I provide a theoretical description of the transform and demonstrate properties of the learned representation on both synthetic data and natural videos. Further, the SMT also provides a unifying geometric perspective on the role of simple and complex cells. I propose a version of SMT in which neurons in the two layers correspond to simple and complex cells. This provides a new functional explanation for these cell types: Simple cells can be viewed as representing a discrete sampling of a smooth manifold in the sensor space, while complex cells can be viewed as representing localized smooth linear functions on the manifold. While each individual complex cell pools from a local region, together they tile (Sengupta et al. 2018) the manifold and build an untangled population representation (DiCarlo & Cox 2007), which tends to preserve the identity of the signal while straightening the transformations (Henaff et al. 2017). In the localized SMT, the complex cell layer is learned in an unsupervised manner based on a diffusion process. The results demonstrate that simple and complex cells are emergent properties in a neural system that is optimized to learn the manifold structure of dynamic sensory inputs and is subject to sparse connectivity constraints.

The second contribution is on the discovery of word factors and word embedding visualization. Co-occurrence statistics based word embedding techniques have proved to be very useful in extracting the semantic and syntactic representation of words as low dimensional continuous vectors. This work discovered that dictionary learning can open up these word vectors as a linear combination of more elementary word factors. I demonstrate many of the learned factors have surprisingly strong semantic or syntactic meaning corresponding to the factors previously identified manually by human inspection. Thus dictionary learning provides a powerful visualization tool for understanding word embedding representations. Furthermore, I show that the word factors can help in identifying key semantic and syntactic differences in word analogy tasks and improve upon the state-of-the-art word embedding techniques in these tasks by a large margin.

The third contribution is on a more efficient and effective way to train energy-based models (EBMs) in high-dimensional spaces. Energy-based models assign unnormalized log-probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising score matching (Vincent 2011) has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good results in data denoising (Saremi & Hyvarinen 2019). However, demonstrations of such models in high-quality sample synthesis of high dimensional data were lacking. Recently, (Song & Ermon 2019) have shown that a generative model trained by denoising score matching accomplishes excellent sample synthesis, when trained with data samples corrupted with multiple levels of noise. Both analysis and empirical evidence show that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, a novel EBM trained with multi-scale denoising score matching is proposed. The model exhibits data generation performance comparable to state-of-the-art techniques such as GANs, and sets a new baseline for EBMs. The proposed model also provides density information and performs well in an image inpainting task.

The high levels goals of this thesis are to: understand the neural representation of sound, produce more robust statistical models of natural sound, and develop models for top-down auditory attention. These are three critical concepts in the auditory system. The neural representation of sound should provide a useful representation for building robust statistical models and directing attention. Robust statistical models are necessary for humans to generalize their knowledge from one domain to the plethora of domains in the real world. And attention is fundamental to the perception of sound, allowing one to prioritize information in the raw audio signal.

First, I approach understanding the neural representation of sound using the efficient coding principle and the physiological characteristics of the cochlea. A theoretical model is developed using convolutional filters and leaky-integrate-and-fire (LIF) neurons to model the cochlear transform and spiking code of the auditory nerve. The goal of this model is to explain the distributed phase code of the auditory nerve response but it lays the foundation for much more.

Second, I investigate an algorithm for audio source separation, called deep clustering. Experiments are performed to evaluate it's robustness, and a new neural network architecture is developed to improve robustness. The experiments show that the conventional recurrent neural network performs sub-optimally, and our dilated convolutional neural network improves robustness while using an order of magnitude fewer parameters. This more parsimonious model is a step towards models which are minimally parameterized and generalize well across many domains.

Third, I develop a new algorithm to address the limitations of the previous deep clustering method. This algorithm can extract multiple sources at once from a mixture using an attentional context or bias. It relies on modulating the computation of the bottom-up pathway using a top-down neural signal, which indicates which sources are of interest. A simple idea from the attentional spotlight method is used to do this: to allow for the top-down neural signal to modulate the gain on a set of low level neurons. This computational method demonstrates one way top-down feedback could direct auditory attention in the brain. Interestingly, this method goes beyond neuroscience, it demonstrates that attention can be about more than efficient computation. The experiments show that it resolves one of the main short comings of deep clustering. The model can extract sources from a mixture without knowing the total number of sources in the mixture, unlike deep clustering.

The major contributions of this work are a theoretical model for the auditory nerve response, a more robust neural network architecture for sound understanding, and a novel and powerful model of top-down auditory attention. I hope that the first contribution will be used to build a better understanding of the complex auditory nerve code. The second to build ever more parsimonious and robust models of source separation. And the third to provide a basis for an under-explored research direction which I believe is the most fruitful for building human-level auditory scene analysis, attention-based source separation.

Although we are rarely aware of it, our ability to visually perceive and successfully interact with the world depends on a rapid and carefully orchestrated sequence of eye movements. Roughly three times a second, large high-velocity movements known as saccades drastically alter the spatial and temporal stream of visual input. To investigate the temporal constraints on saccadic eye movements and identify biases in oculomotor behavior, I developed a simple but novel task: rapid alternating saccades (RAS). Human participants are asked to make a series of eye movements back and forth between stationary targets as quickly as possible.

In Chapter 2, I investigate the characteristics of one of the most prominent and well-known biases in eye guidance, inhibition of return. Participants made RAS between two targets or following an “hourglass” pattern in which the same location is only sampled every fourth eye movement. The experiments revealed that both saccade dwell times and secondary saccade characteristics were dramatically altered when the eye returned directly to a previously viewed location. The effects depended on the direction of movement and the angular difference between subsequent saccades. The results further explicate the inhibition of return phenomenon and provide novel insight into motor and attentional constraints governing the rate of sequential eye movements.

Chapter 3 explores interactions between the eye and the hand. Findings indicate that the maximum rate of RAS increases when concurrent and directionally compatible hand movements accompany the eye movements. The increase is the result of shorter dwell times, higher saccade peak velocity, and a decrease in secondary saccade occurrence. The findings occur independently of changes in saccade amplitude or direction. Hand movements in the opposite direction of the eye result in longer dwell times and more frequent secondary saccades. The experiments illustrate the tight temporal coordination between saccades and arm motor systems during sequential movements.

A current limitation in our understanding of the visual system is its function under natural viewing conditions, especially in the context of dynamic, human behavior.While there have been many advances in understanding the spatial response properties of visual neurons in relation to static stimuli such as natural images, understanding of the corresponding temporal properties has been limited by the lack of high-fidelity datasets that document the properties of the signal that reaches the human retina under natural conditions. In this thesis, I describe the design and construction of a mobile tracking device that can re-create the signal present on the retina of a human as they perform everyday tasks. This device is used to collect high fidelity video and tracking data from human subjects performing a set of tasks that sample the everyday human environment and behavioral repertoire. This new dataset makes it possible to characterize the spatiotemporal statistics of natural time-varying signals as they occur on the retina. Here I examine the spatio-temporal power spectrum, which is of interest as a natural scene statistic in part because it is the Fourier transform of the autocorrelation function. In the absence of ego motion (movement of head and body), the spatiotemporal power spectrum of the dynamic environment has similar power-law structure to that previously reported for Hollywood movies. Head and eye motion modulate the spatiotemporal signal, boosting mid- and high-range temporal frequencies, such that the visual input on the retina is nearly whitened. This can be beneficial for reducing signal redundancy and maximizing the use of available bandwidth in the optic nerve. The phase spectrum, which compliments the power spectrum, also carries relevant information about natural image statistics. Despite the strong perceptual signal carried by the classically defined global phase, I show that it has limited utility to differentiate natural images from noise. However, phase congruency, a locally-defined property of the phase, shows marked differences between the distributions of natural images and noise, as well as differences within separate categories of natural scenes. Finally, I explore the relationship between natural signals and the human visual system by optimizing a neural network to carry the most amount of information through a bottleneck inspired by the human optic nerve, while limiting the energy utilized by neural spikes. I show that a previously proposed model exhibits computational instabilities that hinder the use of autodifferentiation software in training this model, and I offer methods of addressing them. I also show that this model can be reformulated with a restructuring of the network, from a single layer model to an autoencoder framework, avoiding computational instabilities altogether. I conclude with a summary of contributions, as well as a discussion of future areas of exploration.

The Locally Competitive Algorithm (LCA) is a recurrent neural network for performing sparse coding and dictionary learning of natural signals. The network itself lacks much of the anatomical structure we see in real brains, but it does include lateral connectivity and iterative, or recurrent, computation. These together produce population nonlinearities that facilitate desirable coding properties, including improved robustness, selectivity, and efficiency when compared to more traditional architectures. When trained on images of natural scenes, the network also has many overlapping response properties with those of biological neurons. This has encouraged scientists to use it as a means to conceptualize theories of neural coding and explore their consequences. Probing the network enables us to observe and test theories that account for neurophysiological phenomena. In this thesis, we will provide motivation for investigating the model, a detailed derivation of the model, an analysis of its response properties, and offer some extensions. The core computational principles in the LCA are distinct from those used in most production artificial neural networks, and we will argue that there is much to be gained from incorporating LCA-like computations in exchange for feed-forward, pointwise nonlinear model neurons.