An important question when developing photon-counting detectors for computed tomography is how to select energy thresholds. In this work thresholds are optimized by maximizing signal-difference-to-noise ratio squared (SDNR2) in an optimally weighted image and signal-to-noise ratio squared (SNR2) in a gadolinium basis image in a silicon-strip detector and a cadmium zinc telluride (CZT) detector, factoring in pileup and imperfect energy response based on real-world detector systems. To investigate to what extent one single set of thresholds could be applied in various imaging tasks, the robustness of optimal thresholds with 2 to 8 bins is examined with the variation of phantom thicknesses, target materials and detector configurations. In contrast to previous studies, the optimal threshold locations do not always increase with increasing attenuation if pileup is included. With respect to the tradeoff between higher SDNR2 or SNR2 and less data, setting optimal thresholds for a 30 cm phantom yields robust SDNR2 and setting optimal thresholds for a 50 cm phantom yields robust SNR2 with 6 to 8 bins in the silicon-strip detector. Furthermore, setting optimal thresholds for a 30 cm phantom yields robust SDNR2 or SNR2 with 6 to 8 bins and a pixel size less than or equal to 0.5 × 0.5 mm2 in the CZT detector.