One of the more surprising findings in visual cognition is the apparent sparsity of our scene percepts. Yet, scene perception also enables planning and navigation, which require a detailed, structured analysis of the scene geometry, including exit locations and the obstacles along the way. Here, we hypothesize that computation of navigational affordances (e.g., paths to an exit) is a “default” task in the mind, and that task induces selective analysis of the scene geometry most relevant to computing these affordances. In an indoor scene setting, we show that observers more readily detect changes if these changes impact shortest paths to visible exits. We show that behavioral detection rates are explained by a new model of attention that makes heterogeneous-precision inferences about the scene geometry, relative to how its different regions impact navigational affordance computation. This work provides a formal window into the contents of our scene percepts.

Humans have an intuitive understanding of music. We can predict the ensuing notes of a melody given the first few notes, but what exactly drives these predictions? Previous research on musical cognition explores probabilistic models of melody perception where a melody's structure can be inferred given its surface. Other research theorizes about “musical forces”, forces that are analogous to how we represent the physical world, and which inform the way we form expectations about music. We propose a single model of melodic expectation that combines both ideas using a structured generative model and sequential Monte Carlo inference. The generative model formalizes these musical forces, and combined with inference, enables predicting the last note of a melody given the beginning notes. This model explains human performance in an existing dataset of melodic predictions. The model explains more variance than its ablations, and suggests an “intuitive physics” basis for melodic expectation.

Here, we test whether the perception of liquids is based on abstract, physics-based representations. To do so, we use a novel behavioral task that requires generalization across scene configurations. We created 30 animations of liquids flowing through and interacting with surfaces and objects. These animations came from six qualitatively different scene configurations and five viscosity levels. During the experiment, participants viewed two liquid animations (side-by-side), each from a different scene configuration, and they rated the similarity of underlying viscosities. We found that not only average similarity ratings are linearly predicted based on true viscosity differences, but also the identity of the scene configuration pairs was uncorrelated to similarity ratings. This successful generalization in human behavior suggests an underlying abstract, physics-based representation of liquids. We also present a new probabilistic model of liquid perception that accounts for behavioral ratings, quantitatively supporting our hypothesis.

Language understanding in the real world occurs through noise — often, lots of noise. What makes language understanding so robust? Here, we address this challenge with a new approach. We cast language understanding as Bayesian inference in a generative model of how world states arise and project to utterances. We develop this model in a case study of action understanding from language input: inferring the goal of an agent in 2D grid worlds from utterances. The generative model provides a prior over agents' goals, a planner that maps these goals to actions, and a ‘language-renderer' that creates utterances from these actions. The generative model also incorporates GPT-2 as a noisy language production model. We invert this process with sequential Monte Carlo. In a behavioral experiment, the resulting model, called the Generative Semantic Transformation Process, explains evolving goal inferences of humans as utterances unfold.

Bodies of water manifest rich physical interactions via non-linear dynamics. Yet, humans can successfully perceive and negotiate such systems in everyday life. Here, we hypothesize that liquid bodies play such an integral role in human life that the mind automatically computes their approximate flow-paths, with attention dynamically deployed to efficiently predict flow trajectories using coarse mental simulation. When viewing animations of liquids flowing through maze-like scenes, we asked participants to detect temporary slowdowns embedded in these animations. This task, without any overt prompt of path or prediction, reveals that detection rates vary with the moment-to-moment changes in coarse flow-path predictions. Critically, coarse predictions better explain trial-level detection rates than a finer-grained alternative, independently of bottom-up salience of slowdowns. This work suggests liquid flow-path prediction as an implicit task in the mind, and introduces rich attentional dynamics as a new window into intuitive physics computations.

The visual system does not require extensive signal in its inputs to compute rich, three-dimensional (3D) shape percepts. Even under highly degraded stimuli conditions, we can accurately interpret images in terms of volumetric objects. What computations support such broad generalization in the visual system? To answer, we exploit two degraded image modalities – silhouettes and two-tone “Mooney” images – alongside regular shaded images. We test two distinct approaches to vision: deep networks for classification and analysis-by-synthesis for scene inference. Deep networks perform substantially sub-human even after training on 18 times more images per category compared to the existing large-scale image sets for object classification. We also present a novel analysis-by-synthesis architecture that infers 3D scenes from images via optimization in a differentiable, physically-based renderer. This model also performs substantially sub-human. Nevertheless, both approaches can explain some of the key behavioral patterns. We discuss the insights these results provide for reverse-engineering visual cognition.

Human scene understanding involves not just localizing objects, but also inferring the latent causal properties that giverise to the scene for instance, how heavy those objects are. These properties can be guessed based on visual features(e.g., material texture), but we can also infer them from how they impact the dynamics of the scene. Furthermore, theseinferences are performed rapidly in response to dynamic, ongoing information. Here we propose a computational frame-work for understanding these inferences, and three models that instantiate this framework. We compare these models tothe evolution of human beliefs about object masses. We find that while peoples judgments are generally consistent withBayesian inference over these latent parameters, the models that best explain human judgments are approximations to thisinference that hold and dynamically update beliefs. An earlier version of this work was published in the proceedings ofCCN 2018 at https://ccneuro.org/2018/proceedings/1091.pdf.