Face perception is a ubiquitous perceptual task that most people easily perform many times a day, beginning in early childhood. The process of extracting meaningful information from a face for tasks such as face identification, gender discrimination, or emotion discrimination, involves making eye movements to different parts of a face. It is known that most of the information for such tasks can be extracted after just a single initial eye movement. However, the efficiency with which that information can be used may be modulated by the internal representation of faces that are stored in our brains for specific tasks. This dissertation explores several aspects of the interaction of the initial eye movement to a face with internal face representations. One aspect is the evaluation of configural processing of faces in the context of a foveated visual system. Another aspect is the effect of natural statistics of facial expressions on the availability of task-relevant information that may be extracted with an initial eye movement to a face. A third aspect is how individual differences in the initial eye movement may shape the development of internal fixation-specific face representations. All of these aspects are investigated using a combination of human psychophysics studies and computational modeling of face perception and eye movements. The data obtained supports a view of the initial eye movement to a face as a highly practiced and consistent behavior that depends on the statistics of the faces that humans are exposed to during different face discrimination tasks. In turn, this behavior may also shape the internal representations of faces in our brains.

Interpretation of gaze direction is an essential ability that allows humans to infer others' intentions, interests, and future actions in daily social interactions. Humans start to respond to direct and avert gazes at 4 months old (Farroni et al., 2002), and show the ability to follow others’ gaze direction at 10 months old (Brooks & Meltzoff, 2005). Studies have shown that gaze perception relies on the integrated effect of the eye region, head orientation, and body posture (Cline, 1967, p. 19; Hietanen, 2002; Langton et al., 2004; Moors et al., 2015; Otsuka, 2014; Pomianowska et al., 2012; Wollaston, 1824). The gaze of others influences both covert (Bayliss et al., 2004a; Driver et al., 1999; Friesen & Kingstone, 1998) and overt attention (Friesen & Kingstone, 2003; Hood et al., 1998; Mansfield et al., 2003; Ricciardelli et al., 2003). Not only eye gaze but head and body posture also orient observers’ attention (Azarian et al., 2017; Bayliss et al., 2004a; Frischen et al., 2007, p. 20; Hietanen, 1999, 2002). Studies have argued that images of gaze induce attention shift exogenously because the attention orientation effect following the gaze direction is preserved even when the gaze direction is uninformative (Driver et al., 1999; Friesen et al., 2004; Friesen & Kingstone, 1998; Ristic et al., 2007). However, unlike typical peripheral exogenous cues (e.g., flashlight) that only sustain a transient attention orientation that lasts less than 200ms (Posner & Cohen, 1984; Theeuwes, 1991a), attention orientation elicited from gaze develops gradually and decays after 500ms since the cue onset (Friesen & Kingstone, 1998; McKee et al., 2007; Müller & Findlay, 1988a).A limitation of the majority of gaze studies is that they typically use simple images of gaze, foveally presented, and include a simple perceptual task. These displays do not incorporate the dynamics of head and body movements, the surrounding real-world contexts in scenes, or more ecological tasks. Less is known about how humans estimate gaze, orient covert attention, and eye movements with real-world scenes that include the dynamics of gazer movements. The moment-to-moment control of eye movements during gaze-following has also not been studied. The overall goal of this dissertation is to increase the understanding of the perception of gaze and the influence of gaze on attention and eye movements in real-world scenes. Chapter I studies how humans and the state-of-art AI model perceive gaze in naturalistic gaze images. By digitally manipulating the contextual information in the scene (the number of gazers and the presence of the gazed target), Chapter I shows that humans have a higher sensitivity to contextual information changes compared to the AI model, and a potential direction for developing AI model with better ability to integrate spatial information in the future. Chapter II-III explores the process of covert attention orientation, microsaccades bias, and overt attention (eye movement) using videos of dynamic gazer behaviors. Both chapters show the importance of head dynamics in orienting attention. Specifically, Chapter II illustrates the importance of the presence of both head and body motion to sustain covert attention for a longer period (500ms), and microsaccades are correlated with behavioral performance. Chapter III shows that valid gaze cues guide eye movements faster and more accurately toward the gazed target (< 2˚) and improve behavioral performance. It also suggests that the smallest gaze cue guidance effect when heads are removed may be due to the reduced available information in headless bodies. Lastly, Chapter IV proceeds to quantify the dynamic changes of gazer’s heads and relate them to the dynamics of eye movements during gaze following. It demonstrates that observers use information in the visual periphery to execute predictive saccades that anticipate the information in the gazer’s head direction by 190-350ms. Observers simultaneously monitor moment-to-moment changes in the gazer’s head velocity to dynamically alter eye movements and re-fixate the gazer (reverse saccades) when the head accelerates before the initiation of the first forward gaze-following saccade. Together, the dissertation extends our understanding of gaze perception in real-life contexts, and its impacts on observers’ covert attention, overt attention, and fine-grained eye movement control with ecologically relevant dynamic gaze behaviors.

Visual search is a critical and pervasive part of our everyday lives. However, the ease of which we perform search can mask its remarkable computational complexity. Search targets are often difficult to detect and embedded in statistically complex backgrounds. To optimize search, human observers often exploit known statistical properties of the visual environment which provide information about target location. Regularities in the spatial organization of the visual environment (e.g. predictive cues) have shown to be one such type of statistical property which can be leveraged to increase search efficiency. Here, a series of three studies examines how well human observers can exploit spatially predictive cues during multi-fixation search. Further exploration focuses on highlighting human flexibility in altering search strategy to enhance perceptual performance, as well as delineating situations in which predictive information in the environment may actually hurt search performance.

Predictive spatial cues have been shown to improve perceptual performance for a variety of tasks, including visual search, under conditions of forced fixation. However, the potential benefits of predictive cues during multi-fixation search are poorly understood. In the first study, we present a letter identification search task, done in the presence and absence of an array of spatial cues which framed potential target locations. We show that human observers direct their eye movements towards cued locations to improve their search performance compared to when the cues are absent. We also develop a foveated eye movement model, which takes into account the diminishing acuity of the human visual system in the periphery, for the task. Model predictions reveal substantial performance benefits via predictive cues, the size of which are much larger than what is seen in human data.

In the second study, we investigate whether human observers will utilize peripheral predictive cues which reside in display regions which have no chance of containing the target (which we call remote cues). In doing so, observers must depart from a commonly used "saccadic targeting" strategy, where eyes are directed to likely target locations. When informed of the predictive nature of the remote cues, observers readily adopt an atypical eye movement strategy which favors non-target locations to enhance task performance. A foveated model which ignores the cues reveals that these performance benefits are a likely result of foveating peripheral predictive information. Interestingly, a version of the remote cue task in which observers were not informed of cue contingencies, reveals that while observers can adopt atypical saccadic strategies to improve performance, they do not readily engage in such behavior without explicit information.

In the final study, we explore a situation in which predictive cues actually hinder search performance. In natural viewing environments and everyday search tasks, predictive cues often do not mark the only target locations, but merely likely ones. We return to the letter identification search task and modify the cues so that they are only partially predictive of target location. With this increased uncertainty, the presence of predictive cues actually leads to worse performance when the target is highly visible, compared to when cues are absent. A control task in which the spatial cues were not predictive rules out the possibility of the effect being driven by exogenous attentional capture, and a foveated eye movement model reveals that cues hindering search performance would not be predicted by a rational observer. Implications for real-life and vocational search are discussed.

In all, we see that human observers are willing and able to utilize predictive elements via intelligent eye movement selection of the environment to enhance search performance. In addition, when cue information is made clear, observers are able to readily adopt even extremely atypical eye movement strategies to optimize performance. However, human efficiency in implementing these strategies often falls short of that predicted by a near-optimal observer, even leading to a decrement in performance in extreme cases. Understanding the interaction between human performance and predictive cues, then, is critical to assessing natural visual search and optimizing vocational and life-critical search displays.

The human visual field is composed of a high acuity region at the center of gaze called the fovea, and

its complement, the visual periphery. Not much is known about the computations and representations

of the visual periphery, as most of the focus in the field of human (and machine) vision is geared towards

foveal vision. Thus, the focus of this thesis will be on understanding the computations performed by

the human visual system in the visual periphery. In doing so, I will begin by modelling the perception

of clutter and how it changes as a function of the behavioural task and point of fixation, developing

a collection of foveated clutter models that enhance non-foveated models. I will then propose a new

metamer model that renders how the information is distorted in the visual field, and what this tells us

about the computations done in the visual periphery. Finally, I will conclude with the design of two

hybrid man-machine collaborative visual search systems that try to overcome the limitations in human

visual search imposed by the visual periphery and observer inefficiencies in terminating exploration.

Many animals and humans process the visual field with varying spatial resolution (foveated vision) and use peripheral processing to make intelligent eye movements and point the fovea to acquire high-resolution information about objects of interest. A foveated architecture results in computationally efficient rapid scene exploration and can result in energy savings for computer vision. A foveated model can also serve as a proxy to identify circumstances in which humans might make an error and as a tool to understand human vision. Foveated architectures have been implemented into previous computer vision models. However, they have not been explored with the latest computer vision architecture transformer networks, which result in better robustness against adversarial attacks and better representation of spatial relationships across the entire image.

We propose foveated computational module for object classification (FoveaTer) and object detection (FST) integrated into the vision transformer architecture. We evaluate FoveaTer’s computational savings and gains in robustness to adversarial attacks relative to a full-resolution model. We used the self-attention weights to optimize the guidance of the model eye movements. We have also investigated using FoveaTer to predict the various behavioral effects of humans. We performed a psychophysics experiment for the scene categorization task and predicted the human categorization performance using the FoveaTer model. Using two additional psychophysics experiments, a forced-fixation mouse recognition experiment to detect mouse in the visual periphery and a visual search experiment to detect mouse using a limited number of fixations, We have also evaluated how the FST model uses contextual information to guide eye movements like humans.

In addition, we trained anthropomorphic CNN models to detect simulated tumors in simulated 3D Digital Breast Tomosynthesis phantoms and compare their performance and errors against that of radiologists. We provide preliminary results on extending the FST model for tumor search in virtual mammograms.

Thus, the contributions of the dissertation are to further the implementation of computational cost savings for computer vision, to predict perceptual errors of humans, and provide a computational tool to study human vision/cognitive science in the wild.

In the field of oncology, three-dimensional volumetric medical images provide radiologists with a detailed visual representation of various anatomical structures that facilitate the early detection and characterization of malignant lesions but at the cost of an increased search space. Recent work (Lago et al., 2021) establishes that human observers rely heavily on peripheral visual processing away from the point of fixation when searching for signals in 3D volumetric images. The searcher’s over-reliance on peripheral vision interacts strongly with how much of the volume they explore and with how much they report they have explored. Specifically, observers under-explore—as determined by the percentage of the volume covered by the Useful Field of View (UFOV)—and overestimate the percentage of volume they explored through self-report measures. Consequently, they miss small signals during the search. This thesis aims to elucidate the psychological factors mediating human under-exploration of 3D volumetric image data. The second thrust of this thesis is to investigate three solutions to mitigate the detrimental impact of under-exploration in 3D images. The first method is a 2D synthetic view of the 3D data that observers can utilize as additional information when performing the 3D search. I establish through behavioral measurements and a computational model simulating foveated vision how the 2D-S guides eye movements to suspicious regions in the 3D volume. In turn, this guidance allows observers to find the small signal that would otherwise be missed without the 2D-S adjunct. The second method involves a different type of search aid, a convolutional neural network, which acts as a computer-aided detection system to assist human observers during the 3D search. Like the 2D-S, it guides eye movements to suspicious regions in a 3D volumetric image that observers would have otherwise not looked at. The last method is inspired by the power of group decision-making. It investigates how combining multiple independent judgments from a group of searchers can lead to more exploration of the search space and a higher chance of detecting the small signal. Together, the body of work herein provides empirical results from laboratory studies to further our understanding of how humans interact with 3D imaging modalities with the goal of improving healthcare services relating to early cancer screenings.

Covert attention refers to the ability to attend to parts of an image without moving the eye. Cues and contexts predictive of a target’s location can orient covert attention and improve perceptual performance. The last four decades have resulted in several theories of covert attention, attributing these performance benefits to limited resources, zoom, spotlights, or a weighting of visual information. However, it is difficult to map these concepts to neuronal populations. The aim of this thesis is to propose and validate feedforward Convolutional Neural Networks (CNNs) with no inbuilt resource constraints, explicit attention mechanisms, or feedback connections, as models of covert attention and study their emergent behavioral and neuronal properties. This thesis is broadly divided into two parts, with the first and second parts focusing on behavior and neuronal properties respectively. In part one (Chapter II), we benchmark the CNNs against theoretically optimal Bayesian Ideal Observer models (BIO) and human psychophysics data from eighteen human observers each on three classic covert attention tasks: Posner cueing, visual search, and contextual cueing. We show that the CNNs learn from images to utilize predictive cues and contexts in all three tasks and capture the behavioral signatures shown by human subjects. These results hold for several variations of the tasks, including central and peripheral pre-cues, noise levels, reaction time measures, discrimination tasks, and network sizes. Both the BIO and CNNs capture human signatures of covert attention but the CNN achieves a better fit across tasks. These results indicate that the human behavioral signatures on the three tasks might be a consequence of task optimization without the need to assume resource limitations. Part two (Chapters III and IV) of the thesis focuses on the neuronal properties of the CNNs discussed in part one for the Posner cueing and contextual cueing to understand why these behavioral signatures emerge in the network. While neurophysiological studies of attention typically manipulate attention via predictive cues and observe their effect on neurons tuned to targets, how and where in the visual hierarchy the cue itself is detected, processed, and integrated with the target information is not well-understood. We carry out a system-wide analysis of 1.9 million neurons for the Posner task and 1.1 million neurons for the contextual cueing task to help understand this. In both tasks, we find that the earlier layers are retinotopic, separately tuned to the target and cue/configuration, while the deeper layers can integrate the target information with the cue/configuration information. The influence of the cue/configuration on the target responses increases on going deeper into the network in both tasks, consistent with neurophysiology. In the Posner task, we find three neuronal mechanisms of cueing: a BIO-like cue-weighted integration across locations, opponency across locations, and interaction with the ReLU activation function. We present a set of testable neuronal properties, several of which are consistent with known properties from neurophysiological studies of attention. In the contextual cueing task, we further benchmark the networks on six task variations from the literature and find good agreement with previous results. These chapters establish a system-wide characterization of the network computations that mediate the behavioral signatures of covert attention. Together, we benchmark the behavioral and neuronal properties expected of neuronal populations learning to optimize task accuracy on covert attention tasks, develop a toolkit to understand neural networks trained on simple psychophysics tasks, and provide a set of testable hypotheses which can potentially help infer attention mechanisms from neuronal data.

Although the facilitation of visual search by contextual information is well established, there is little understanding of the independent contributions of different types of contextual cues in scenes. Additional work in quantifying the time course of the influence of various contextual cues has not been performed, nor have researchers investigated how cue information is extractable across the visual field. Here we manipulated three types of spatial contextual information: object co-occurrence, multiple object configurations, and scene gist. We measured the spatial informativeness of each cue to target localization and isolated the benefits of each contextual cue to target detectability, its impact on decision bias, and the guidance of eye movements. To assess how cues are combined, we compare observed sensitivity during detection with multiple cues to a theoretical optimal combination of the various cues. We also utilize a novel paradigm where scene viewing time was contingent upon the number of fixations within a scene made by observers. To assess observers’ ability to extract cue information across the visual field, we observed their performance at detecting cues in scenes shown exclusively in the visual periphery.

We find that object-based information guides eye movements and facilitates perceptual judgments more than background information. Despite its relatively weaker influence on search behavior, background information is shown to be most easily extracted across the visual field and likely to support the parsing of multiple object configurations in scenes. Multiple object configuration information specifically is implicated in the guidance of initial search, providing coarse guidance that is later localized further by co-occurring object information. The degree of guidance and facilitation of each contextual cue can be related to its contribution to reducing the spatial uncertainty about the target location as measured by human explicit judgments about likely target locations. Comparison of target detectability across various cue conditions suggests that the performance improvements with multiple cues are consistent with an optimal integration of independent cues.

In addition to exploring influences of spatial cues on visual search task performance, we were also interested in assessing a non-spatial cue’s effect on target detection performance and eye movement guidance. We manipulated the scale of a target object relative to its surroundings and found that observers were significantly worse at detecting mis-scaled targets relative to normally sized targets. Unsurprisingly, this non-spatial cue did not have as dramatic an effect on eye movement guidance as the three spatial cues. However, this emphasizes the importance of considering non-spatial scene information as a possible contextual influencer of human behavior.

Overall, our results improve the understanding of the interplay of distinct contextual scene components and their contributions to search guidance, a landmark behavior that differentiates human and biological vision from basic machine vision. The results are also useful in informing the type of information that might improve computer-based object detection and scene understanding.