Model-free (MF) and model-based (MB) reinforcement learn-ing (RL) have provided a successful framework for under-standing both human behavior and neural data. These two sys-tems are usually thought to compete for control of behavior.However, it has also been proposed that they can be integratedin a cooperative manner. For example, the Dyna algorithm usesMB replay of past experience to train the MF system, and hasinspired research examining whether human learners do some-thing similar. Here we introduce an approach that links MFand MB learning in a new way: via the reward function. Givena model of the learning environment, dynamic programmingis used to iteratively approximate state values that monotoni-cally converge to the state values under the optimal decisionpolicy. Pseudorewards are calculated from these values andused to shape the reward function of a MF learner in a waythat is guaranteed not to change the optimal policy. We showthat this method offers computational advantages over Dyna intwo classic problems. It also offers a new way to think aboutintegrating MF and MB RL: that our knowledge of the worlddoesn’t just provide a source of simulated experience for train-ing our instincts, but that it shapes the rewards that those in-stincts latch onto. We discuss psychological phenomena thatthis theory could apply to, including moral emotions.

How do we learn to think better, and what can we do to pro-mote such metacognitive learning? Here, we propose that cog-nitive growth proceeds through metacognitive reinforcementlearning. We apply this theory to model how people learn howfar to plan ahead and test its predictions about the speed ofmetacognitive learning in two experiments. In the first experi-ment, we find that our model can discern a reward structure thatpromotes metacognitive reinforcement learning from one thathinders it. In the second experiment, we show that our modelcan be used to design a feedback mechanism that enhancesmetacognitive reinforcement learning in an environment thathinders learning. Our results suggest that modeling metacog-nitive learning is a promising step towards promoting cognitivegrowth.

What is the optimal way to make a decision given that yourtime is limited and your cognitive resources are bounded? Toanswer this question, we formalized the bounded optimal de-cision process as the solution to a meta-level Markov deci-sion process whose actions are costly computations. We ap-proximated the optimal solution and evaluated its predictionsagainst human choice behavior in the Mouselab paradigm,which is widely used to study decision strategies. Our compu-tational method rediscovered well-known heuristic strategiesand the conditions under which they are used, as well as novelheuristics. A Mouselab experiment confirmed our model’smain predictions. These findings are a proof-of-concept thatoptimal cognitive strategies can be automatically derived as therational use of finite time and bounded cognitive resources.

People’s cognitive strategies are jointly shaped by function and computational constraints. Resource-rational analysis lever- ages these constraints to derive rational models of people’s cognitive strategies from the assumption that people make rational use of limited cognitive resources. We present a resource-rational analysis of planning and evaluate its predic- tions in a newly developed process tracing paradigm. In Ex- periment 1, we find that a resource-rational planning strategy predicts the process by which people plan more accurately than previous models of planning. Furthermore, in Experiment 2, we find that it also captures how people’s planning strategies adapt to the structure of the environment. In addition, our ap- proach allows us to quantify for the first time how close peo- ple’s planning strategies are to being resource-rational and to characterize in which ways they conform to and deviate from optimal planning.