The cross-modal literature posits a weak-to-strong continuum of synesthesia. One extreme views cross-modal associations as idiosyncratic and unique to synesthetes. The other extreme suggests that cross-modal associations follow a general pattern across individuals, and are mediated by emotional associations. We tested these views by examining differences between music-color synesthetes and non-synesthetes in their consistency of color associations and memory for music. We find that music-color associations follow the same general pattern across these groups. A two-dimensional mapping is found to mode (major/minor) and tempo. Slow-minor music (thought to convey sadness) is associated with blue, fast-minor with red (anger), fast-major with yellow (happiness), and slow-major with green (calmness). Both groups are consistent in their associations over time, and synesthesia has no effect on memory. We conclude that music-color synesthesia may be an extension of normal psychological processes that govern cross-modal associations, with individuals aligning music and color based on emotional congruence.