We study the set of infinite volume ground states of Kitaev's quantum double model
on $\mathbb{Z}^2$ for an arbitrary finite abelian group $G$. It is known that these models
have a unique frustration-free ground state. Here we drop the requirement of frustration
freeness, and classify the full set of ground states. We show that the ground state space
decomposes into $|G|^2$ different charged sectors, corresponding to the different types of
abelian anyons (also known as superselection sectors). In particular, all pure ground
states are equivalent to ground states that can be interpreted as describing a single
excitation. Our proof proceeds by showing that each ground state can be obtained as the
weak$^*$ limit of finite volume ground states of the quantum double model with suitable
boundary terms. The boundary terms allow for states which represent a pair of excitations,
with one excitation in the bulk and one pinned to the boundary, to be included in the
ground state space.