We study how the intrinsic anomalous Hall conductivity is modified in two-dimensional crystals with broken time-reversal symmetry due to weak inhomogeneity of the applied electric field. Focusing on a clean noninteracting two-band system without band crossings, we derive the general expression for the Hall conductivity at small finite wave vector q to order q^{2}, which governs the Hall response to the second gradient of the electric field. Using the Kubo formula, we show that the answer can be expressed through the Berry curvature, Fubini-Study quantum metric, and the rank-3 symmetric tensor which is related to the quantum geometric connection and physically corresponds to the gauge-invariant part of the third cumulant of the position operator. We further compare our results with the predictions made within the semiclassical approach. By deriving the semiclassical equations of motion, we reproduce the result obtained from the Kubo formula in some limits. We also find, however, that the conventional semiclassical description in terms of the definite position and momentum of the electron is not fully consistent because of singular terms originating from the Heisenberg uncertainty principle. We thus present a clear example of a case when the semiclassical approach inherently suffers from the uncertainty principle, implying that it should be applied to systems in nonuniform fields with extra care.