People have strong intuitions about which colors do and do not match particular pieces of music — a phenomenon often conceptualized through semantic mediation. We further explored the specific strategies people employ to navigate these color-music associations which offers crucial insights to identifying the cognitive mechanisms that enable such cross-domain associations. We show that while some people rely more on intuition, other people actively seek justification of association by consulting common linguistic descriptors, emotional contents, and environmental cues. We found that more spontaneous strategies lessen the role of semantic mediation, while more evaluative strategies, especially those involving the use of language, amplify it. Notably, the use of evaluative strategies introduced an asymmetrical effect for matching and mismatching colors. Additionally, individuals employing similar strategies associated a given music excerpt with more similar colors, suggesting that strategy alignment enhances the consistency of color-music associations. Interestingly, this pattern of convergence was not observed among individuals who predominantly relied on guessing.