Vasopressin signaling has important effects on the regulation of social behaviors and stress responses, and is considered a promising pathway to target for new therapeutics of stress-induced psychiatric disorders. Although there is evidence for sex differences in the behavioral effects of arginine vasopressin (AVP), few data have directly compared the effects of stress on endogenous AVP signaling in males and females. We used California mice (Peromyscus californicus) to study the short and long term effects of social defeat stress on AVP immunoreactive cells in the paraventricular nucleus (PVN) and the posteromedial bed nucleus of the stria terminalis (BNSTmp). Acute exposure to defeat increased AVP/c-fos cells in the PVN and SON of both males and females. In contrast, there were sex differences in the long term effects of defeat. Males but not females exposed to defeat had less avp mRNA in the PVN, and in two experiments defeat reduced the number of AVP positive cells in the caudal PVN of males but not females. Interestingly, during relatively benign social encounters with a target mouse, there was a rapid decrease in AVP percent staining (including cell bodies and fibers) in the PVN of males but not females. Defeat reduced AVP percent staining in males, but did not block the socially induced decrease in percent staining. When mice were tested in resident-intruder tests, males exposed to defeat were no less aggressive than control males whereas aggression was abolished in females. However, bouts of aggression were positively correlated with the number of AVP neurons in the BNSTmp of control males but not stressed males, suggesting that different mechanisms mediate aggression in control and stressed males. These data show that while acute AVP responses to defeat are similar in males and females, the long term effects of defeat on AVP are stronger in males.