Fast magnetic fluctuations due to thermal torques have useful technological functionality ranging from cryptography to probabilistic computing. The characteristic time of fluctuations in typical uniaxial anisotropy magnets studied so far is bounded from below by the well-known energy relaxation mechanism. This time scales as $\alpha^{-1}$, where $\alpha$ parameterizes the strength of dissipative processes. Here, we theoretically analyze the fluctuating dynamics in easy-plane and antiferromagnetically coupled nanomagnets. We find in such magnets, the dynamics are strongly influenced by fluctuating intrinsic fields, which give rise to an additional dephasing-type mechanism for washing out correlations. In particular, we establish two time scales for characterizing fluctuations (i) the average time for a nanomagnet to reverse|which for the experimentally relevant regime of low damping is governed primarily by dephasing and becomes independent of $\alpha$, (ii) the time scale for memory loss of a single nanomagnet|which scales as $\alpha^{-1/3}$ and is governed by a combination of energy dissipation and dephasing mechanism. For typical experimentally accessible values of intrinsic fields, the resultant thermal-fluctuation rate is increased by multiple orders of magnitude when compared with the bound set solely by the energy relaxation mechanism in uniaxial magnets. This could lead to higher operating speeds of emerging devices exploiting magnetic fluctuations.