In this paper, we propose a lightweight wearable thermoelectric (TE) cooler with a rationally designed flexible heatsink. Heatsinks are commonly designed for use with stationary applications, and are consequently rigid and heavy. These traditional heatsinks are incompatible with wearable applications, which must be durable, mechanically flexible, and lightweight while maintaining performance. This paper presents a flexible heatsink based on a ternary composite of silicone elastomer, phase-change material, and graphite powder; this combination is needed to achieve high flexibility and durability as well as the optimum heat capacity and thermal conductivity. Those factors are key requirements for a heatsink to optimize the cooling performance of a TE cooler and maintain a longer cooling capacity. With our optimized ternary composite flexible heatsink, we achieved a cooling temperature of ∼5 K at 0.5 W input power and kept cooling for more than 5 h under ambient conditions. On-body testing of the wearable TE cooler with flexible heatsink was also performed to demonstrate potential applications in the real-world. Our work provides fundamental insights to designing wearable TE devices and paves the way for innovative solutions for on-body thermal management applications such as clothing, hats, seat cushions, and other portable devices.