Membrane-mediated particle interactions depend both on the properties of the particles themselves and the membrane environment in which they are suspended. Experiments have shown that chiral rod-like inclusions dissolved in a colloidal membrane of opposite handedness assemble into colloidal rafts, which are finite-sized reconfigurable droplets consisting of a large but precisely defined number of rods. We systematically tune the chirality of the background membrane and find that, in the achiral limit, colloidal rafts acquire complex structural properties and interactions. In particular, rafts can switch between 2 chiral states of opposite handedness, which alters the nature of the membrane-mediated raft-raft interactions. Rafts with the same chirality have long-ranged repulsions, while those with opposite chirality acquire attractions with a well-defined minimum. Both attractive and repulsive interactions are qualitatively explained by a continuum model that accounts for the coupling between the membrane thickness and the local tilt of the constituent rods. These switchable interactions enable assembly of colloidal rafts into intricate higher-order architectures, including stable tetrameric clusters and "ionic crystallites" of counter-twisting domains organized on a binary square lattice. Furthermore, the properties of individual rafts, such as their sizes, are controlled by their complexation with other rafts. The emergence of these complex behaviors can be rationalized purely in terms of generic couplings between compositional and orientational order of fluids of rod-like elements. Thus, the uncovered principles might have relevance for conventional lipid bilayers, in which the assembly of higher-order structures is also mediated by complex membrane-mediated interactions.