Educational video games have the potential to be used as assessments of student understanding of complex concepts. However, the interpretation of the rich stream of complex data that results from the tracking of in-game actions is so difficult that it is one of the most serious blockades to the use of educational video games or simulations to assess student understanding, and there is currently no systematic approach to extracting relevant data from log files. This study attempts to determine whether data mining techniques can be used to extract information from log files that allows for the formation of testable hypotheses. The log files in this study come from an educational video game teaching students about the identification of fractions. The three data mining techniques used in this study were: cluster analysis, sequence mining, and classification.
Cluster analysis was used to examine the individual actions each student took in a given attempt to solve each game level. This led to the identification of valid solution strategies students used and mathematical errors and game-related errors students made as they tried to solve game levels. Sequence mining was used to examine the change in strategy use as each student moved through the game levels. This led to the identification of strategy sequences representing different paths students took to arrive at the correct solution. Classification was used to examine the change in the number of attempts each student required to solve the levels in each stage. This led to the identification of performance trajectories representing improvement, decline, or lack of change in performance over time.
To demonstrate the usefulness of the extracted information and provide initial evidence that the interpretation of the extracted information was valid, testable hypotheses from the results of each data mining technique were generated examining whether the grouping of students resulting from each of the three data mining techniques differed significantly on paper-and-pencil pretest scores, posttest scores, or the gain in scores between pretest and posttest. The groups resulting from each of the three data mining techniques differed significantly on pretest and posttest scores, with students in groups interpreted as representing lower performance demonstrating lower performance on the tests and students in groups interpreted as representing higher performance demonstrating higher performance on the tests. However, none of the three techniques led to the identification of groups of students that differed significantly on the gain in scores between pretest and posttest.
