Residential racial segregation in the U.S. has been hypothesized as a fundamental cause driving health disparities across racial groups. Potential mechanisms include economic and social marginalization, subsequent constrained opportunities, and high stress. Yet evidence on residential segregation's association with health among Black and White children-particularly longitudinally-is sparse. This study aims to address this gap. We used data from the Panel Study of Income Dynamics (PSID), a national longitudinal study of U.S. households, analyzing information on 1,251 Black and 1,427 White children who participated in the Child Development Supplement (CDS) at least twice (survey waves 1997, 2002, 2007, 2014). We fit individual fixed-effects models to estimate the within-person association of neighborhood-level residential segregation, measured with local Getis-Ord G* statistics, with three outcomes (general health, weight status, and behavioral problems). We examined heterogeneous effects by age and sex. We also examined associations between health and childhood segregation trajectories, i.e., the pattern of children's residential segregation exposures from birth through when their health outcomes were measured, providing additional insight on dynamic experiences of segregation. In fixed effects models, among Black children, higher segregation was associated with worse self-rated health, especially for Black children who were older (aged 11-17 years). In trajectory models, among White children, moving out of highly segregated neighborhoods was associated with a lower probability of poor self-rated health, while moving into those neighborhoods or back and forth between neighborhood types were both associated with increased behavioral problems. Our findings highlight the importance of early-life residential segregation in shaping persistent racial health disparities, as well as the costs of segregation for all children living in highly segregated neighborhoods.