This study developed and validated a measure that captures variation in common local idioms of distress and mental health problems experienced by women in South Sudan, a country which has experienced over 50 years of violence, displacement, and political, social, and economic insecurity. This measure was developed during a randomized controlled trial of the Adolescent Girls Initiative (AGI) and used qualitative Free Listing (n = 102) and Key Informant interviews (n = 27). Internal reliability and convergent validity were assessed using data from 3,137 randomly selected women (ages 14-47) in 100 communities in South Sudan. Test-retest and inter-rater reliability were assessed using responses from 180 women (ages 15-58) who completed the measure once, and 129 of whom repeated the measure an average of 12 days (SD = 8.3) later. Concurrent validity was assessed through the ratings of 22 AGI leaders about the presence or absence of mental health symptoms in the 180 women in the test-retest sample. The study resulted in the development of the South Sudan Mental Health Assessment Scale, a 24-item measure assessing six idioms of distress. The scale consisted of one factor and had excellent internal, test-retest, and interrater reliability. The scale also demonstrated good convergent and concurrent validity and performed well psychometrically. Moreover, its development provides an example for other organizations, working in environments where mental health measures have not yet been developed and validated, to create and validate measures relevant to their populations. In this way, the role of mental health in development settings can be more rapidly assessed.