Skip to main content
eScholarship
Open Access Publications from the University of California

Bridging the Measurement Gap: a Large Language Model Method of Assessing Open-Ended Question Complexity

Creative Commons 'BY' version 4.0 license
Abstract

Question-asking, an essential yet understudied activity, holds significant implications for fields such as learning, creativity, and cognitive development. The quality, and complexity in particular, of the questions are recognized as crucial factors affecting these fields. Previous research explored question complexity through Bloom's taxonomy, but measurement remains challenging. Recent advancements have enabled automated scoring of psychological tasks but have not been applied to open-ended question complexity. Here, we address this gap by employing large language model (LLM) techniques to predict human ratings of open-ended question complexity. Our results reveal that our LLM-generated complexity scores correlated strongly with human complexity ratings in both the holdout-responses (r = .73) and holdout-item set (r = .77), whilst also exceeding baseline methods tested. The research emphasizes the significance of LLMs in psychological research and their potential in automating question complexity assessment. This study also highlights exciting possibilities for usage of LLMs in education and psychology.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View