Proceedings of the Annual Meeting of the Cognitive Science Society

We propose an agent model consisting of perceptual and pro-prioceptive pathways. It actively samples a sequence of per-cepts from its environment using the perception-action loop.The model predicts to complete the partial percept and propri-ocept sequences observed till each sampling instant, and learnswhere and what to sample from the prediction error, withoutsupervision or reinforcement. The model is exposed to twokinds of stimuli: images of fully-formed handwritten numer-als/alphabets, and videos of gradual formation of numerals.For each object class, the model learns a set of salient locationsto attend to in images and a policy consisting of a sequence ofeye fixations in videos. Behaviorally, the same model givesrise to saccades while observing images and tracking whileobserving videos. The proposed agent is the first of its kindto interact with and learn end-to-end from static and dynamicenvironments to generate realistic handwriting with state-of-the-art performance.