Sinking particulate organic carbon out of the surface ocean sequesters carbon on decadal to millennial timescales. Predicting the particulate carbon flux is therefore critical for understanding both global carbon cycling and the future climate. Microbes play a crucial role in particulate organic carbon degradation, but the impact of depth-dependent microbial dynamics on ocean-scale particulate carbon fluxes is poorly understood. Here we scale-up essential features of particle-associated microbial dynamics to understand the large-scale vertical carbon flux in the ocean. Our model provides mechanistic insight into the microbial contribution to the particulate organic carbon flux profile. We show that the enhanced transfer of carbon to depth can result from populations struggling to establish colonies on sinking particles due to diffusive nutrient loss, cell detachment, and mortality. These dynamics are controlled by the interaction between multiple biotic and abiotic factors. Accurately capturing particle-microbe interactions is essential for predicting variability in large-scale carbon cycling.