Brain-derived neurotrophic factor (BDNF) Val66Met polymorphism may be an important source of heterogeneity seen in cognitive aging, although the specific relationship between this polymorphism and cognition remains controversial and may depend on the sex of participants. We assessed 2668 older black and white adults and fit linear mixed models to digit symbol substitution test (DSST) performance assessed in years 0 (baseline), 4, 7, and 9 to examine the interaction between sex and BDNF genotype on the intercept (i.e., estimated baseline DSST) and change in DSST over 9 years, adjusted for covariates. Sex interacted with BDNF genotype to predict DSST intercept (F[1,1599] = 7.4, p < 0.01) and 9-year change (F[1,1183] = 4.1, p = 0.04) in white participants only. Initially, white male Val/Val carriers had lower DSST scores (37.6, SE = 0.8) in comparison with male Met carriers (difference, -1.7; 95% CI, -3.2 to -0.3) and female Val/Val carriers (difference, -5.6; 95% CI, -6.8 to -4.3). White female Met carriers showed a slower rate of change (annual rate of change = -0.6, SE = 0.1) in comparison with female Val/Val carriers (difference, -0.2; 95% CI, -0.4 to -0.02) and male Met carriers (difference, -0.3; 95% CI, -0.5 to -0.02). Our findings suggest that BDNF Val66Met and sex should be considered in future endeavors aimed at treating or preventing neurodegenerative disorders.