Carbon markets play an important role in firms and governments climate strategies. Carbon crediting mechanisms allow project developers to earn carbon credits through mitigation projects. Several studies have raised concerns about environmental integrity, though a systematic evaluation is missing. We synthesized studies relying on experimental or rigorous observational methods, covering 14 studies on 2346 carbon mitigation projects and 51 studies investigating similar field interventions implemented without issuing carbon credits. The analysis covers one-fifth of the credit volume issued to date, almost 1 billion tons of CO2e. We estimate that less than 16% of the carbon credits issued to the investigated projects constitute real emission reductions, with 11% for cookstoves, 16% for SF6 destruction, 25% for avoided deforestation, 68% for HFC-23 abatement, and no statistically significant emission reductions from wind power and improved forest management projects. Carbon crediting mechanisms need to be reformed fundamentally to meaningfully contribute to climate change mitigation.