Recent evidence indicates that color categories can exert astrong influence over color matching in both perception andmemory. We explore this phenomenon by analyzing the costfunction for perceptual error. Our analysis is developed withinthe mathematical framework of rate–distortion theory. Ac-cording to our approach, the goal of perception is to minimizethe expected cost of error while subject to a strong constrainton the capacity of perceptual processing. We propose that thecost function in color perception is defined by the sum of twocomponents: a metric cost associated with the magnitude of er-ror in color space, and a cost associated with perceptual errorsthat cross color category boundaries. A computational modelembodying this assumption is shown to produce an excellent fitto empirical data. The results generally suggest that what ap-pear as ‘errors’ in working memory performance may reflectreasonable and systematic behaviors in the context of costs.