Impaired face recognition for certain face categories is known in both humans and non-human primates. A previous study found that chimpanzees are worse at discriminating infant faces than adult faces. Chimpanzee infant faces are different from adult faces in color and shape. However, it remains unclear whether impaired face discrimination for infant faces is solely due to facial color or shape, or due to a combination of both. We investigated which facial features have greater effects on the difficulty of face identification. Adult chimpanzees were required to match the faces in a matching-to-sample task with four types of face stimuli whose shape and color are manipulated independently. We found that chimpanzees’ performance decreased when asked to match the faces with infant coloration regardless of the shape. This study is the first to demonstrate the impairment effect of infantile coloration on face recognition in non-human primates.