Changes in the function of inhibitory interneurons (INs) during cortical development could contribute to the pathophysiology of neurodevelopmental disorders. Using all-optical in vivo approaches, we find that parvalbumin (PV) INs and their immature precursors are hypoactive and transiently decoupled from excitatory neurons in postnatal mouse somatosensory cortex (S1) of Fmr1 KO mice, a model of fragile X syndrome (FXS). This leads to a loss of parvalbumin INs (PV-INs) in both mice and humans with FXS. Increasing the activity of future PV-INs in neonatal Fmr1 KO mice restores PV-IN density and ameliorates transcriptional dysregulation in S1, but not circuit dysfunction. Critically, administering an allosteric modulator of Kv3.1 channels after the S1 critical period does rescue circuit dynamics and tactile defensiveness. Symptoms in FXS and related disorders could be mitigated by targeting PV-INs.