Cognitive scientists often conduct experiments that require a single participant to complete multiple – often hundreds of – repetitions of the same task (i.e., experiments with multiple trials). By having many observations per participant, these studies improve the measurement properties of the outcome they are designed to assess. However, repeating the same task results in systematic changes in cognitive processes over time that could also affect the reliability and validity of the outcome measure. To date, there is no modelling framework that can easily capture these systematic cognitive changes across multiple, independent trials. Response time models are an increasingly popular way of inferring cognitive processes such as caution, information processing efficiency, and bias, but they assume that these processes do not change over the course of an experiment. In this study, we extended a popular response time model, the Diffusion Decision Model, to be able to capture systematic changes in its parameters across trials. We focused on two processes that should vary in almost all multi-trial experiments: information processing efficiency (increase over time as people get better at the task) and caution (decrease over time as people learn the task demands). Using model comparison methods on pre-existing data, we showed that both processes varied systematically across time for a large proportion of participants, even when the experiment that generated the data did not explicitly intend to manipulate either of those processes.

Participants wore a smartphone, which collected GPS, audio,accelerometry and image data, in a pouch around their necksfor a period of two weeks. After a retention interval of oneweek, they were asked to judge the specific day on whicheach of a selection of images was taken. To account forpeople’s judgements, we proposed a mixture model of fourprocesses - uniform guessing, a signal detection process basedon decaying memory strength, a week confusion process anda event confusion process in which the sensor streams wereused to calculate the similarity of events. A model selectionexercise testing all possible subsets of the processes favoureda model that included only the event confusion model. GPSsimilarities were found to be the most significant predictors,followed by audio and accelerometry similarities and thenimage similarities.

Most data analyses rely on models. To complement statistical models, psychologists have developed cognitive models, which translate observed variables into psychologically interesting constructs. Response time models, in particular, assume that response time and accuracy are the observed expression of latent variables including 1) ease of processing, 2) response caution, 3) response bias, and 4) non-decision time. Inferences about these psychological factors, hinge upon the validity of the models' parameters. Here, we use a blinded, collaborative approach to assess the validity of such model-based inferences. Seventeen teams of researchers analyzed the same 14 data sets. In each of these two-condition data sets, we manipulated properties of participants' behavior in a two-alternative forced choice task. The contributing teams were blind to the manipulations, and had to infer what aspect of behavior was changed using their method of choice. The contributors chose to employ a variety of models, estimation methods, and inference procedures. Our results show that, although conclusions were similar across different methods, these "modeler's degrees of freedom" did affect their inferences. Interestingly, many of the simpler approaches yielded as robust and accurate inferences as the more complex methods. We recommend that, in general, cognitive models become a typical analysis tool for response time data. In particular, we argue that the simpler models and procedures are sufficient for standard experimental designs. We finish by outlining situations in which more complicated models and methods may be necessary, and discuss potential pitfalls when interpreting the output from response time models.