Interpretation of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) serosurveillance studies is limited by poorly defined performance of antibody assays over time in individuals with different clinical presentations. We measured antibody responses in plasma samples from 128 individuals over 160 days using 14 assays. We found a consistent and strong effect of disease severity on antibody magnitude, driven by fever, cough, hospitalization, and oxygen requirement. Responses to spike protein versus nucleocapsid had consistently higher correlation with neutralization. Assays varied substantially in sensitivity during early convalescence and time to seroreversion. Variability was dramatic for individuals with mild infection, who had consistently lower antibody titers, with sensitivities at 6 months ranging from 33 to 98% for commercial assays. Thus, the ability to detect previous infection by SARS-CoV-2 is highly dependent on infection severity, timing, and the assay used. These findings have important implications for the design and interpretation of SARS-CoV-2 serosurveillance studies.