Analysis of the specificity and kinetics of neutralizing antibodies (nAbs) elicited by SARS-CoV-2 infection is crucial for understanding immune protection and identifying targets for vaccine design. In a cohort of 647 SARS-CoV-2-infected subjects, we found that both the magnitude of Ab responses to SARS-CoV-2 spike (S) and nucleoprotein and nAb titers correlate with clinical scores. The receptor-binding domain (RBD) is immunodominant and the target of 90% of the neutralizing activity present in SARS-CoV-2 immune sera. Whereas overall RBD-specific serum IgG titers waned with a half-life of 49 days, nAb titers and avidity increased over time for some individuals, consistent with affinity maturation. We structurally defined an RBD antigenic map and serologically quantified serum Abs specific for distinct RBD epitopes leading to the identification of two major receptor-binding motif antigenic sites. Our results explain the immunodominance of the receptor-binding motif and will guide the design of COVID-19 vaccines and therapeutics.