The magnetic properties of the quasi-two-dimensional van der Waals magnet Fe5-δGeTe2 (F5GT), which has a high ferromagnetic ordering temperature TC∼315 K, remain to be better understood. It has been demonstrated that the magnetization of F5GT is sensitive to both the Fe deficiency δ and the thermal-cycling history. Here, we investigate the structural and magnetic properties of F5GT single crystals with a minimal Fe deficiency (|δ|≤0.1), utilizing combined x-ray and neutron scattering techniques. Our study reveals that the quenched F5GT single crystals experience an irreversible, first-order transition at TS∼110 K upon first cooling, where the stacking order partly or entirely converts from ABC-stacking order to AA-stacking order. Importantly, the magnetic properties, including the magnetic moment direction and the enhanced TC after the thermal cycling, are intimately related to the alteration of the stacking order. Our work highlights the significant influence of the lattice symmetry to the magnetism in F5GT.