We compared the associations of diabetes mellitus (DM) and other cardiovascular disease (CVD) risk factors with decline in the ankle-brachial index (ABI) over 4 years in participants with and without peripheral artery disease (PAD). A total of 566 participants, 300 with PAD, were followed prospectively for 4 years. Mean (SD) baseline ABI values were 0.70 (0.13) for participants with both PAD and DM, 0.67 (0.14) for participants with only PAD, 1.10 (0.13) for participants with only DM, and 1.10 (0.10) for participants with neither PAD nor DM. After adjusting for age, sex, and baseline ABI, the corresponding ABI change from baseline to 4-year follow-up were -0.02, -0.04, +0.05, and +0.05, respectively. Compared to participants with neither PAD nor DM, participants with only PAD showed significantly more ABI decline (p <0.01), while the decline in participants with both PAD and DM was borderline non-significant (p = 0.06). After adjustments for baseline ABI, age, sex, African American ethnicity, and other CVD risk factors, independent factors associated with ABI decline in participants with PAD in the lower ABI leg were older age and elevated D-dimer. DM was not related to ABI decline. Despite being an important risk factor for PAD, DM was not independently associated with ABI decline. This could reflect the effect of DM promoting both PAD and lower-extremity arterial stiffness, resulting in a small decline in the ABI over time. In conclusion, ABI change over time in persons with diabetes may not accurately reflect underlying atherosclerosis.