Little is known about the relationship between perceived and objective measures of the built environment and physical activity behavior among rural populations. Within the context of a lifestyle-change intervention trial for rural women, Strong Hearts, Healthy Communities (SHHC), we examined: (1) if Walk Score (WS), an objective built environment measure, was associated with perceived built environment (PBE); (2) if WS and PBE were associated with moderate-to-vigorous physical activity (MVPA); and (3) if MVPA changes were modified by WS and/or PBE. Accelerometers and questionnaires were used to collect MVPA and PBE. Bivariate analyses and linear mixed models were used for statistical analyses. We found that WS was positively associated with perceived proximity to destinations (p < 0.001) and street shoulder availability (p = 0.001). MVPA was generally not associated with WS or PBE. Compared to controls, intervention group participants increased MVPA if they lived in communities with the lowest WS (WS = 0), fewer perceived walkable destinations, or extremely safe perceived traffic (all p < 0.05). Findings suggest that WS appears to be a relevant indicator of walkable amenities in rural towns; results also suggest that the SHHC intervention likely helped rural women with the greatest dearth of built environment assets to improve MVPA.