Estrogen and progesterone have been extensively studied in the mammary gland, but the molecular effects of androgen remain largely unexplored. Transgender men are recorded as female at birth but identify as male and may undergo gender-affirming androgen therapy to align their physical characteristics and gender identity. Here we perform single-cell-resolution transcriptome, chromatin, and spatial profiling of breast tissues from transgender men following androgen therapy. We find canonical androgen receptor gene targets are upregulated in cells expressing the androgen receptor and that paracrine signaling likely drives sex-relevant androgenic effects in other cell types. We also observe involution of the epithelium and a spatial reconfiguration of immune, fibroblast, and vascular cells, and identify a gene regulatory network associated with androgen-induced fat loss. This work elucidates the molecular consequences of androgen activity in the human breast at single-cell resolution.