During interaction with others, we perceive and produce so-cial actions in close temporal distance or even simultaneously.It has been argued that the motor system is involved in bothprocesses, but how does it distinguish in this processing be-tween self and other? In this paper, we present a model ofself-other distinction within a hierarchical sensorimotor sys-tem that is based on principles of perception-action couplingand predictive processing. For this we draw on mechanisms as-sumed for the integration of cues to generate sense of agency,i.e., the sense that an action is self-generated. We report re-sults from simulations of different social scenarios, showingthat the model is able to solve the problem of the dual use ofthe sensorimotor system.