Limited knowledge is available on the relationship between antigen-specific immune responses and COVID-19 disease severity. We completed a combined examination of all three branches of adaptive immunity at the level of SARS-CoV-2-specific CD4⁺ and CD8⁺ T cell and neutralizing antibody responses in acute and convalescent subjects. SARS-CoV-2-specific CD4⁺ and CD8⁺ T cells were each associated with milder disease. Coordinated SARS-CoV-2-specific adaptive immune responses were associated with milder disease, suggesting roles for both CD4⁺ and CD8⁺ T cells in protective immunity in COVID-19. Notably, coordination of SARS-CoV-2 antigen-specific responses was disrupted in individuals ≥ 65 years old. Scarcity of naive T cells was also associated with aging and poor disease outcomes. A parsimonious explanation is that coordinated CD4⁺ T cell, CD8⁺ T cell, and antibody responses are protective, but uncoordinated responses frequently fail to control disease, with a connection between aging and impaired adaptive immune responses to SARS-CoV-2.