According to the theory of common coding, actions are represented in terms of their sensory effects. Hence, performing or anticipating an action biases perception. Previous studies provided evidence for this notion by showing how the perception of ambiguous visual stimuli can be affected by concurrent actions. Here we investigated whether performing a directed action can affect the perception of ambiguous auditory stimuli in a gamified dual-task. In an online study, participants had to avoid obstacles in an endless runner game while classifying the pitch shift in an ambiguous sequence of Shepard tones. Response times indicate interference between both tasks, but pitch shift classifications seem to be unaffected by the motor task. Meanwhile, participants showed a strong compatibility effect between base pitch and pitch shift classifications, in line with a typical SMARC effect. We discuss possible reasons for the absence of perceptual modulations and implications for common coding approaches.

We introduce a modular recurrent neural architecture, which learns distributed, generative temporal models of bio-logical motion. It encodes modal visual and proprioceptive (angular) biological motions separately by means of autoencoders,structuring respective postures, motion directions, and motion magnitudes separately. The submodal encoders are interdepen-dent by predicting each other’s next autoencoder states temporally. As a result, distributed attractor states can develop fromself-generated motions. We show that the architecture is able to synchronize its activities across modalities towards overallconsistent action-encoding attractors. Moreover, the developing spatial and temporal structures can complete partially observ-able actions, e.g., when only providing visual information. Furthermore, we show that the network is capable of simulatingwhole-body actions without any sensory stimulation, thus imagining unfolding actions. Finally, we show that the network isable to infer the visual perspective on a biological motion. Thus, the neural architecture enables embodied perspective takingand action inference.

Theories of event-predictive, anticipatory behavior controlsuggest that complex action planning and control is segmentedinto sequences of anticipated subgoals and according behav-ioral events, which accomplish the subgoals. Here we focus onthe cognitive dynamics during successive subgoal activations.We combined a virtual object interaction task (prehension andtransport of a bottle) with a crossmodal congruency task. An-ticipatory crossmodal congruency effects (aCCEs) occur at thegoal of the current behavior, before the goal is reached. TheseaCCEs appear to be stronger during prehension, while visualdistractors at the currently irrelevant movement target have noeffect. While the results so far provide only partial supportfor the proposed anticipatory, sequential control process, theparadigm is well-suited to probe the dynamic changes of spa-tial body representations in object interactions.

According to theories of anticipatory behavior control, actionplanning and control is realized by activating desired goalstates. From an event-predictive perspective, this activationshould focus sensorimotor processing on expected, upcomingevent boundaries. Previous studies have shown that periper-sonal hand space (PPHS) is remapped to the future hand lo-cation in a grasping task before the movement commences.Here, we investigated if the current hand posture interfereswith the anticipatory remapping of PPHS. Participants had tograsp virtual bottles from two differently oriented starting pos-tures. During the prehension, they received a vibrotactile stim-ulus on their right index finger or on their thumb, while a vi-sual stimulus appeared at the bottle, either matching the futurefinger position, or not. Participants had to name the stimu-lated finger. While the hand posture affected verbal responsetimes, the anticipatory remapping remained unchanged. Ap-parently, the predictive processes that realize the anticipatoryremapping, generalize over initial hand postures.

According to theories of event-predictive cognition, neural processing focuses on the next relevant interaction targets. Evidence for this notion comes from the anticipatory crossmodal congruency effect (aCCE), which implies that spatial body representations are mapped onto future goal locations in advance of a goal-directed action. Here we present a free energy based normative process model that accounts for the aCCE quantitatively by applying crossmodal mappings between vision and touch as well as active inference. A comparison with a diffusion model shows that our model accounts for the response time distributions and the aCCE with a sparser set of parameters. However, the temporal dynamics of the model require further fine tuning to account for all aspects of the aCCE. The model shows how the free energy framework can be used to account for behavioral data in general and how to implement theories of event-predictive cognition in a normative cognitive process model.

Learning to write is characterized by bottom-up mimicking of characters and top-down writing from memory. We introduce a CNN-RNN model that implements both pathways: It can (i) directly write a letter by generating a motion trajectory given an image, (ii) first classify the character in the image and then determine its motion trajectory `from memory', or (iii) use a combination of both pathways. The results show that, in one-shot and few-shot learning, the model profits from different combinations of the pathways: The generation of different character variants works best when the top-down is supported by the bottom-up pathway. Refilling occluded images of efficiently learned characters works best when using the top-down pathway alone. Overall, the architecture implies that a weighted merge of bottom-up and top-down information into a latent, generative code fosters the development of compositional encodings, which can be reused in efficient learning tasks.

According to most recent theories of multisensory integration,weighting of different modalities depends on the reliability ofthe involved sensory estimates. Top-down modulations havebeen studied to a lesser degree. Furthermore, it is still debatedwhether working memory maintains multisensory informationin a distributed modal fashion, or in terms of an integrated rep-resentation. To investigate whether multisensory integrationis modulated by task relevance and to probe the nature of theworking memory encodings, we combined an object interac-tion task with a size estimation task in an immersive virtualreality. During the object interaction, we induced multisen-sory conflict between seen and felt grip aperture. Both, visualand proprioceptive size estimation showed a clear modulationby the experimental manipulation. Thus, the results suggestthat multisensory integration is not only driven by reliability,but is also biased by task demands. Furthermore, multisensoryinformation seems to be represented by means of interactivemodal representations.

We demonstrate that inference-based goal-directed behavior can be done by utilizing the temporal gradients in re-current neural network (RNN). The RNN learns a dynamic sensorimotor forward model. Once the RNN is trained, it can beused to execute active-inference-based, goal-directed policy optimization. The internal neural activities of the trained RNNessentially model the predictive state of the controlled entity. The implemented optimization process projects the neural activ-ities into the future via the RNN recurrences following a tentative sequence of motor commands (encoded in neurons akin torecurrent parametric biases). This sequence is adapted by back-projecting the error between the forward-projected hypotheticalstates and desired (goal-like) system states onto the motor commands. Few cycles of forward projection and goal-based errorbackpropagation yield the sequences of motor commands that control the dynamical systems. As an example, we show that atrained RNN model can be used to effectively control a quadrocopter-like system.