Refugee communities are vulnerable to housing insecurity, which drives numerous health disparity outcomes in a historically marginalized population. The COVID-19 pandemic has only worsened the ongoing affordable housing crisis in the United States while continuing to highlight disparities in health outcomes across populations. We conducted interviewer-administered surveys with refugee and asylum seekers in San Diego County at the height of the COVID-19 pandemic to understand the social effects and drivers of COVID-19 in one of the largest refugee communities in the United States. Staff from a community-based refugee advocacy and research organization administered the surveys from September-November 2020. 544 respondents participated in the survey, which captured the diversity of the San Diego refugee community including East African (38%), Middle Eastern (35%), Afghan (17%), and Southeast Asian (11%) participants. Nearly two-thirds of respondents (65%) reported living in overcrowded conditions (> 1 individual per room) and 30% in severely crowded conditions (> 1.5 individuals per room). For each additional person per room, self-reported poor emotional health increased. Conversely, family size was associated with a lower likelihood of reporting poor emotional health. Crowded housing was significantly associated with a lower probability of accessing a COVID-19 diagnostic test, with every additional reported person per room there was approximately an 11% increase in the probability of having never accessed a COVID-19 testing. Access to affordable housing had the largest effect size and was associated with fewer people per room. Overcrowding housing is a structural burden that reduces COVID-19 risk mitigation behaviors. Improved access to affordable housing units or receiving vouchers could reduce overcrowded housing in vulnerable refugee communities.