Sensory inputs control motor behavior with a strength, or gain, that can be modulated according to the movement conditions. In smooth pursuit eye movements, the response to a brief perturbation of target motion is larger during pursuit of a moving target than during fixation of a stationary target. As a step toward identifying the locus and mechanism of gain modulation, we test whether it acts on signals that are in visual or motor coordinates. Monkeys tracked targets that moved at 15°/s in one of eight directions, including left, right, up, down, and the four oblique directions. In eight-ninths of the trials, the target underwent a brief perturbation that consisted of a single cycle of a 10 Hz sine wave of amplitude ±5°/s in one of the same eight directions. Even for oblique directions of baseline target motion, the magnitude of the eye velocity response to the perturbation was largest for a perturbation near the axis of target motion and smallest for a perturbation along the orthogonal axis. Computational modeling reveals that our data are reproduced when the strength of visual-motor transmission is modulated in sensory coordinates, and there is a static motor bias that favors horizontal eye movements. A network model shows how the output from the smooth eye movement region of the frontal eye fields (FEF(SEM)) could implement gain control by shifting the peak of a visual population response along the axes of preferred image speed and direction.