Optical nonlinearities are key resources in the contemporary photonics toolbox, relevant to quantum gate operations and all-optical switches. Chemical modification is often used to control the nonlinear response of materials at the microscopic level, but on-the-fly manipulation of such response is challenging. Tunability of optical nonlinearities in the mid-infrared (IR) is even less developed, hindering its applications in chemical sensing or IR photonic circuitry. Here, we report control of vibrational polariton coherent nonlinearities by manipulation of macroscopic parameters such as cavity longitudinal length or molecular concentration. Further two-dimensional IR investigations reveal that nonlinear dephasing provides the dominant source of the observed ultrafast polariton nonlinearities. The reported phenomena originate from the nonlinear macroscopic polarization stemming from strong coupling between microscopic molecular excitations and a macroscopic photonic cavity mode.