We have investigated the effect of partial isovalent anion substitution in Ga1-xMnxAs on electrical transport and ferromagnetism. Substitution of only 2.4percent of As by P induces a metal-insulator transition at a constant Mn doping of x=0.046 while the replacement of 0.4 percent As with N results in the crossover from metal to insulator for x=0.037. This remarkable behavior is consistent with a scenario in which holes located within an impurity band are scattered by alloy disorder in the anion sublattice. The shorter mean free path of holes, which mediate ferromagnetism, reduces the Curie temperature TC from 113 K to 60 K (100 K to 65 K) upon the introduction of 3.1 percent P (1percent N) into the As sublattice.