Solid-state high harmonic generation (sHHG) spectroscopy is an emerging ultrafast technique for studying key material properties such as electronic structure at and away from equilibrium. sHHG anisotropy measurements, where sHHG spectra are recorded depending on the driving electric field relative to the crystal lattice, have become a powerful tool for studying crystal symmetries. Previous works on two-dimensional materials and other quantum materials have often used substrate-supported samples, assuming that all sHHG signals originate from the sample due to the relatively large bandgap of the substrate. While this assumption is generally reasonable, we show that some sHHG emissions from commonly used substrates can occur at moderate intensities of the sHHG driving field. In addition, we show that it is essential to consider not only the sHHG yield from a substrate but also its angular dependence relative to the material of interest. Specifically, in this work, the power-dependent and polarization angle-resolved sHHG emissions of fused silica, calcium fluoride, diamond, and sapphire of two different crystalline qualities and orientations are compared using a mid-infrared (MIR) driving field. This empirical characterization aims to guide the substrate selection for sHHG studies of novel materials to minimize the misattribution and interference of substrate-related sHHG emissions, which opens the possibility to study a wider array of materials.