Working memory is a critical building block for almost all cognitive tasks, and impairment can cause significant disruption to daily life routines. We investigated the functional connectivity (FC) of the visuo-spatial working memory network in temporal lobe epilepsy and its relationship to the underlying white matter tracts emanating from the hippocampus. Fifty-two patients with unilateral hippocampal sclerosis (HS) (30 left) and 30 healthy controls underwent working memory functional MRI (fMRI) and Diffusion Tensor Imaging (DTI). Six seed regions were identified for FC analysis; 4 within a task-positive network (left and right middle frontal gyri and superior parietal lobes), and 2 within a task-negative network (left and right hippocampi). FC maps were created by extracting the time-series of the fMRI signal in each region in each subject and were used as regressors of interest for additional GLM fMRI analyses. Structural connectivity (SC) corresponding to areas to which the left and right hippocampi were connected was determined using tractography, and a mean FA for each hippocampal SC map was calculated. Both left and right HS groups showed atypical FC between task-positive and task-negative networks compared to controls. This was characterised by co-activation of the task-positive superior parietal lobe ipsilateral to the typically task-negative sclerosed hippocampus. Correlational analysis revealed stronger FC between superior parietal lobe and ipsilateral hippocampus, was associated with worse performance in each patient group. The SC of the hippocampus was associated with the intra-hemispheric FC of the superior parietal lobe, in that greater SC was associated with weaker parieto-frontal FC. The findings suggest that the segregation of the task-positive and task-negative FC networks supporting working memory in TLE is disrupted, and is associated with abnormal structural connectivity of the sclerosed hippocampus. Co-activation of parieto-temporal regions was associated with poorer working memory and this may be associated with working memory dysfunction in TLE.