We describe and analyze the performance of metric learning systems, including deep neural networks (DNNs), on anew dataset of human similarity judgments of Fribbles, naturalistic, part-based objects. Metrics trained using pixel-based or DNN-based representations fail to explain our experimental data, but a metric trained with a viewpoint-invariant,part-based representation produces a good fit. We also find that although neural networks can learn to extract the part-based representation—and therefore should be capable of learning to model our data—networks trained with a triplet lossfunction based on similarity judgments do not perform well. We analyze this failure, providing a mathematical descriptionof the relationship between the metric learning objective function and the triplet loss function. The comparatively poorperformance of neural networks appears to be due to the nonconvexity of the optimization problem in network weightspace. We discuss the implications for neural network research as a whole.

In the past few years, deep convolutional neural networks(CNNs) trained on large image data sets have shown impres-sive visual object recognition performances. Consequently,these models have attracted the attention of the cognitive sci-ence community. Recent studies comparing CNNs with neuraldata from cortical area IT suggest that CNNs may—in addi-tion to providing good engineering solutions—provide goodmodels of biological visual systems. Here, we report evidencethat CNNs are, in fact, not good models of human visual per-ception. We show that a 3D shape inference model explainshuman performance on an object shape similarity task betterthan CNNs. We argue that deep neural networks trained onlarge amounts of image data to maximize object recognitionperformance do not provide adequate models of human vision.

Research in cognitive science and machine learning suggests that learning systems can use small subsets of valuabletraining items in order to quickly learn to achieve good task performance. We hypothesize that people often use smallsubsets of stored exemplars to quickly train new neural networks, called extemporaneous networks, when faced with tasksfor which they do not currently have dedicated networks. We explore this hypothesis using participants’ responses in abehavioral experiment to identify easy versus difficult training items. We find that a network confidence measure indicatesa network trained with a small set of good items provides the best account of participants’ reaction times. Furthermore,computer simulations demonstrate that learning systems can achieve good performance when trained with small sets ofeasy exemplars. Our results indicate that humans may complete tasks using extemporaneously-created networks trainedinternally on small datasets.

Performance in estimating the depth and shape of an ellipse on the basis of stereo, motion, and vergence angle information was compared for three models of visual depth cue combination. The three models were a weak model (strict modularity, with no interaction between motion and stereo cues), a modified weak model (restricted interaction allowed between motion and stereo cues), and a strong model (unconstrained interaction between all visual cues). Results are that the modified weak model performed best overall indicating that its structure, which contains both modular and interactive features, has advantages over both the extreme modular organization of the weak model and the extreme interactive organization of the strong model. In addition, the different weighting of motion and stereo cues by the modified weak model in the depth and shape judgment tasks provides a motivation for multiple visual representations of three-dimensional space.