Cognitive processing slows with age. We sought to determine the importance of white matter integrity, assessed by diffusion tensor imaging (DTI), at influencing cognitive processing speed among normal older adults, assessed using a novel battery of computerized, non-verbal, choice reaction time tasks. We studied 131 cognitively normal adults aged 55-87 using a cross-sectional design. Each participant underwent our test battery, as well as MRI with DTI. We carried out cross-subject comparisons using tract-based spatial statistics. As expected, reaction time slowed significantly with age. In diffuse areas of frontal and parietal white matter, especially the anterior corpus callosum, fractional anisotropy values correlated negatively with reaction time. The genu and body of the corpus callosum, superior longitudinal fasciculus, and inferior fronto-occipital fasciculus were among the areas most involved. This relationship was not explained by gray or white matter atrophy or by white matter lesion volume. In a statistical mediation analysis, loss of white matter integrity mediated the relationship between age and cognitive processing speed.