INTRODUCTION: The institutions (i.e., hubs) making up the National Institutes of Health (NIH)-funded network of Clinical and Translational Science Awards (CTSAs) share a mission to turn observations into interventions to improve public health. Recently, the focus of the CTSAs has turned increasingly from translational research (TR) to translational science (TS). The current NIH Funding Opportunity Announcement (PAR-21-293) for CTSAs stipulates that pilot studies funded through the CTSAs must be focused on understanding a scientific or operational principle underlying a step of the translational process with the goal of developing generalizable solutions to accelerate translational research. This new directive places Pilot Program administrators in the position of arbiters with the task of distinguishing between TR and TS projects. The purpose of this study was to explore the utility of a set of TS principles set forth by NCATS for distinguishing between TR and TS. METHODS: Twelve CTSA hubs collaborated to generate a list of Translational Science Principles questions. Twenty-nine Pilot Program administrators used these questions to evaluate 26 CTSA-funded pilot studies. RESULTS: Factor analysis yielded three factors: Generalizability/Efficiency, Disruptive Innovation, and Team Science. The Generalizability/Efficiency factor explained the largest amount of variance in the questions and was significantly able to distinguish between projects that were verified as TS or TR (t = 6.92, p < .001) by an expert panel. CONCLUSIONS: The seven questions in this factor may be useful for informing deliberations regarding whether a study addresses a question that aligns with NCATS vision of TS.