Vortices are fundamental magnetic topological structures characterized by a curling magnetization around a highly stable nanometric core. The control of the polarization of this core and its gyration is key to the utilization of vortices in technological applications. So far polarization control has been achieved in single-material structures using magnetic fields, spin-polarized currents or spin waves. Here we demonstrate local control of the vortex core orientation in hybrid structures where the vortex in an in-plane Permalloy film coexists with out-of-plane maze domains in a Co/Pd multilayer. The vortex core reverses its polarization on crossing a maze domain boundary. This reversal is mediated by a pair of magnetic singularities, known as Bloch points, and leads to the transient formation of a three-dimensional magnetization structure: a Bloch core. The interaction between vortex and domain wall thus acts as a nanoscale switch for the vortex core polarization.