The impacts of climate extremes on terrestrial ecosystems are poorly understood but important for predicting carbon cycle feedbacks to climate change. Coupled climate-carbon cycle models typically assume that vegetation recovery from extreme drought is immediate and complete, which conflicts with the understanding of basic plant physiology. We examined the recovery of stem growth in trees after severe drought at 1338 forest sites across the globe, comprising 49,339 site-years, and compared the results with simulated recovery in climate-vegetation models. We found pervasive and substantial "legacy effects" of reduced growth and incomplete recovery for 1 to 4 years after severe drought. Legacy effects were most prevalent in dry ecosystems, among Pinaceae, and among species with low hydraulic safety margins. In contrast, limited or no legacy effects after drought were simulated by current climate-vegetation models. Our results highlight hysteresis in ecosystem-level carbon cycling and delayed recovery from climate extremes.