We have an amazing ability to categorize objects in the world around us. Nevertheless, how cortical regions in human ventral temporal cortex (VTC), which is critical for categorization, support this behavioral ability, is largely unknown. Here, we examined the relationship between neural responses and behavioral performance during the categorization of morphed silhouettes of faces and hands, which are animate categories processed in cortically adjacent regions in VTC. Our results reveal that the combination of neural responses from VTC face- and body-selective regions more accurately explains behavioral categorization than neural responses from either region alone. Furthermore, we built a model that predicts a person's behavioral performance using estimated parameters of brain-behavior relationships from a different group of people. Moreover, we show that this brain-behavior model generalizes to adjacent face- and body-selective regions in lateral occipitotemporal cortex. Thus, while face- and body-selective regions are located within functionally distinct domain-specific networks, cortically adjacent regions from both networks likely integrate neural responses to resolve competing and perceptually ambiguous information from both categories.