Individuals can use both linguistic and non-linguistic features of the speech signal to identify talkers. For instance, listeners have more difficulty identifying talkers in unfamiliar languages compared to a native language (language familiarity effect), implying that language-specific knowledge aids talker identification. In the present study, the source of the language familiarity benefit on talker identification was investigated as listeners identified talkers in their native language as well as non-native languages. Experiment 1 was designed to explore the influence of L2 proficiency on talker identification across languages. Experiment 2 further investigated individual differences in L1 phonetic perception and their contribution to talker identification by comparing English listeners’ performance across different language conditions that varied in the availability of linguistic cues. Results imply that familiarity with a specific language (L1 or L2) did not explain individual variation in language familiarity effect. Rather, in addition to the native language benefit, talker identification may be supported by general sensitivity to sound structures in language, modulated by the availability of higher-level linguistic information