The total angular momentum of a closed system is a conserved quantity, which should remain constant in time for any excitation experiment once the pumping signal has been extinguished. Such conservation, however, is never satisfied in practice in any real-time first-principles description of the demagnetization process. Furthermore, there is growing experimental evidence that the same takes place in experiments. The missing angular momentum is usually associated to lattice vibrations, which are not measured experimentally and are never considered in real-time simulations. Here we critically analyze the issue and conclude that current state-of-the-art simulations violate angular momentum conservation already at the electronic level of description. This shortcoming originates from an oversimplified description of the spin-orbit coupling, which includes atomic contributions but neglects completely that of itinerant electrons. We corroborate our findings with time-dependent simulations using model tight-binding Hamiltonians, and show that indeed such conservation can be reintroduced by an appropriate choice of spin-orbit coupling. The consequences of our findings on recent experiments are also discussed.