A large number of studies have been published examining the implications of climate change for agricultural productivity that, broadly speaking, can be divided into process-based modeling and statistical approaches. Despite a general perception that results from these methods differ substantially, there have been few direct comparisons. Here we use a data-base of yield impact studies compiled for the IPCC Fifth Assessment Report (Porter et al 2014) to systematically compare results from process-based and empirical studies. Controlling for differences in representation of CO2 fertilization between the two methods, we find little evidence for differences in the yield response to warming. The magnitude of CO2 fertilization is instead a much larger source of uncertainty. Based on this set of impact results, we find a very limited potential for on-farm adaptation to reduce yield impacts. We use the Global Trade Analysis Project (GTAP) global economic model to estimate welfare consequences of yield changes and find negligible welfare changes for warming of 1 °C-2 °C if CO2 fertilization is included and large negative effects on welfare without CO2. Uncertainty bounds on welfare changes are highly asymmetric, showing substantial probability of large declines in welfare for warming of 2 °C-3 °C even including the CO2 fertilization effect.