We present the first measurement of cross-correlation between the lensing potential, reconstructed from cosmic microwave background (CMB) polarization data, and the cosmic shear field from galaxy shapes. This measurement is made using data from the Polarbear CMB experiment and the Subaru Hyper Suprime-Cam (HSC) survey. By analyzing an 11 deg2 overlapping region, we reject the null hypothesis at 3.5σ and constrain the amplitude of the cross-spectrum to , where is the amplitude normalized with respect to the Planck 2018 prediction, based on the flat Λ cold dark matter cosmology. The first measurement of this cross-spectrum without relying on CMB temperature measurements is possible owing to the deep Polarbear map with a noise level of ∼6 μK arcmin, as well as the deep HSC data with a high galaxy number density of . We present a detailed study of the systematics budget to show that residual systematics in our results are negligibly small, which demonstrates the future potential of this cross-correlation technique.