The theory of a plasma undulator excited by a short intense laser pulse in a parabolic plasma channel is presented. The undulator fields are generated either by the laser pulse incident off-axis and/or under the angle with respect to the channel axis. Linear plasma theory is used to derive the wakefield structure. It is shown that the electrons injected into the plasma wakefields experience betatron motion and undulator oscillations. Optimal electron beam injection conditions are derived for minimizing the amplitude of the betatron motion, producing narrow-bandwidth undulator radiation. Polarization control is readily achieved by varying the laser pulse injection conditions.