The origin of the Moon's large-scale topography is important for understanding lunar geology, lunar orbital evolution, and the Moon's orientation in the sky. Previous hypotheses for its origin have included late accretion events, large impacts, tidal effects, and convection processes. However, testing these hypotheses and quantifying the Moon's topography is complicated by the large basins that have formed since the crust crystallized. Here we estimate the low-order lunar topography and gravity spherical harmonics outside these basins and show that the bulk of the degree-2 topography is consistent with a crust-building process controlled by early tidal heating throughout the Moon. The remainder of the degree-2 topography is consistent with a frozen tidal-rotational bulge that formed later, at a semi-major axis of »32 Earth radii. The probability of the degree-2 shape having these two separate tidal characteristics by chance is less than 1%. We also infer that internal density contrasts eventually reoriented the Moon's polar axis 36 ± 4°, to the present configuration we observe today. Together, these results link the geology of the near and far sides, and resolve long-standing questions about the Moon's low-order shape, gravity, and history of polar wander.