The latency of visual responses generally decreases as contrast increases. Recording in the lateral geniculate nucleus (LGN), we find that response latency increases with increasing contrast in ON cells for some visual stimuli. We propose that this surprising latency trend can be explained if ON cells rest further from threshold at higher contrasts. Indeed, while contrast changes caused a combination of multiplicative gain change and additive shift in LGN cells, the additive shift predominated in ON cells. Modeling results supported this theory: the ON cell latency trend was found when the distance-to-threshold shifted with contrast, but not when distance-to-threshold was fixed across contrasts. In the model, latency also increases as surround-to-center ratios increase, which has been shown to occur at higher contrasts. We propose that higher-contrast full-field stimuli can evoke more surround inhibition, shifting the potential further from spiking threshold and thereby increasing response latency.