Insights into symbiosis between eukaryotic hosts and their microbiomes have shifted paradigms on what determines host fitness, ecology, and behavior. Questions remain regarding the roles of host versus environment in shaping microbiomes, and how microbiome composition affects host fitness. Using a model system in ecology, phytoplankton, we tested whether microbiomes are host-specific, confer fitness benefits that are host-specific, and remain conserved in time in their composition and fitness effects. We used an experimental approach in which hosts were cleaned of bacteria and then exposed to bacterial communities from natural environments to permit recruitment of microbiomes. We found that phytoplankton microbiomes consisted of a subset of taxa recruited from these natural environments. Microbiome recruitment was host-specific, with host species explaining more variation in microbiome composition than environment. While microbiome composition shifted and then stabilized over time, host specificity remained for dozens of generations. Microbiomes increased host fitness, but these fitness effects were host-specific for only two of the five species. The shifts in microbiome composition over time amplified fitness benefits to the hosts. Overall, this work solidifies the importance of host factors in shaping microbiomes and elucidates the temporal dynamics of microbiome compositional and fitness effects.