How do representations of space inform our perception of time? In language, spatial vocabulary is frequently used to describe temporal concepts, and spatial information biases temporal perception even in non-verbal tasks. In contrast, temporal information typically exerts little, if any, influence on the perception of spatial extent. Here, we used spatial and temporal reproduction tasks, both with and without verbal and spatial dual tasks, to investigate the mechanism underlying the asymmetric relation between space and time. Specifically, we tested whether the asymmetric interference between spatial and temporal stimulus attributes arises from interference in verbal or visuospatial working memory. We found that loading visuospatial working memory, but not verbal working memory, eliminated the asymmetric pattern of interference. This suggests that the interference between spatial and temporal representations arises due to processing constraints in visuospatial working memory. These findings are discussed in terms of the load theory of attention and the relative automaticity with which spatial and temporal information is processed.