Experimental observations indicate that the human immunodeficiency virus (HIV) can replicate in both the follicular and extrafollicular parts of secondary lymphoid tissues, HIV's primary replication site. The former is an immune privileged zone with low cytotoxic lymphocyte activity, and thus HIV replication is primarily concentrated there. Mathematical models and stochastic Gillespie simulations show that this compartmentalization potentially explains several seemingly counterintuitive observations. First, the observation of post-therapy viral decline rate's independence of cytotoxic T-lymphocyte (CTL) presence in simian immunodeficiency virus (SIV)-infected macaques, assuming CTL-mediated lysis significantly contributes to viral suppression. Second, the slow emergence of CTL-escape mutants during chronic infection even if CTL-mediated lysis is responsible for viral suppression. Heterogeneity in CTL activity, and consequently the selection pressure and infected cell population sizes, between the follicular and extrafollicular compartments can explain these findings. The effect of secondary lymphoid tissue compartmentalization is also examined in the context of non-escape scenarios, where the mutant HIV strain has a different infectivity to the "wild" strain, but is recognised by the CTL response. These findings highlight the importance of measuring viral populations separately in the extrafollicular and follicular compartments; using viral load in peripheral blood as an observable hides the heterogeneity between compartments that might be responsible for the particular patterns seen in the dynamics and evolution of HIV in vivo.