We are very familiar with certain objects; we can quickly rec-ognize our cars, friends and collaborators despite heavy occlu-sion, unusual lighting, or extreme viewing angles. We can alsodetermine if two very different views of a stranger are indeedof the same person. How can we recognize familiar objectsquickly, while performing deliberate, perceptual inference onunfamiliar objects? We describe a model combining an iden-tity classification network for familiar faces with an analysis bysynthesis approach for unfamiliar faces to make rich inferencesabout any observed face. We additionally develop an onlinenon-parametric clustering algorithm for recognition of repeat-edly experienced unfamiliar faces, and show how new facescan become familiar by being consolidated into the identityrecognition network. Finally, we show that this model predictshuman behavior in viewpoint generalization and identity clus-tering tasks, and predicts processing time differences betweenfamiliar and unfamiliar faces.