INTRODUCTION: Top-down control underlies our ability to attend relevant stimuli while ignoring irrelevant, distracting stimuli and is a critical process for prioritizing information in working memory (WM). Prior work has demonstrated that top-down biasing signals modulate sensory-selective cortical areas during WM, and that the large-scale organization of the brain reconfigures due to WM demands alone; however, it is not yet understood how brain networks reconfigure between the processing of relevant versus irrelevant information in the service of WM. METHODS: Here, we investigated the effects of task goals on brain network organization while participants performed a WM task that required participants to detect repetitions (e.g., 0-back or 1-back) and had varying levels of visual interference (e.g., distracting, irrelevant stimuli). We quantified changes in network modularity-a measure of brain sub-network segregation-that occurred depending on overall WM task difficulty as well as trial-level task goals for each stimulus during the task conditions (e.g., relevant or irrelevant). RESULTS: First, we replicated prior work and found that whole-brain modularity was lower during the more demanding WM task conditions compared to a baseline condition. Further, during the WM conditions with varying task goals, brain modularity was selectively lower during goal-directed processing of task-relevant stimuli to be remembered for WM performance compared to processing of distracting, irrelevant stimuli. Follow-up analyses indicated that this effect of task goals was most pronounced in default mode and visual sub-networks. Finally, we examined the behavioral relevance of these changes in modularity and found that individuals with lower modularity for relevant trials had faster WM task performance. DISCUSSION: These results suggest that brain networks can dynamically reconfigure to adopt a more integrated organization with greater communication between sub-networks that supports the goal-directed processing of relevant information and guides WM.