There is a growing interest in the mode-by-mode understanding of electron and phonon transport for improving energy conversion technologies, such as thermoelectrics and photovoltaics. Whereas remarkable progress has been made in probing phonon-phonon interactions, it has been a challenge to directly measure electron-phonon interactions at the single-mode level, especially their effect on phonon transport above cryogenic temperatures. Here we use three-pulse photoacoustic spectroscopy to investigate the damping of a single sub-terahertz coherent phonon mode by free charge carriers in silicon at room temperature. Building on conventional pump-probe photoacoustic spectroscopy, we introduce an additional laser pulse to optically generate charge carriers, and carefully design temporal sequence of the three pulses to unambiguously quantify the scattering rate of a single-phonon mode due to the electron-phonon interaction. Our results confirm predictions from first-principles simulations and indicate the importance of the often-neglected effect of electron-phonon interaction on phonon transport in doped semiconductors.