Background

Evaluation of patients' health care experiences is central to measuring patient-centred care. However, different instruments tend to be used at the hospital or departmental level but rarely both, leading to a lack of standardization of patient experience measures.

Objective

To validate the Consumer Quality Index (CQI) Inpatient Hospital Care for use on both department and hospital levels.

Design

Using cross-sectional observational data, we investigated the internal validity of the questionnaire using confirmatory factor analyses (CFA), and the generalizability of the questionnaire for use at the department and hospital levels using generalizability theory.

Setting and participants

22924 adults hospitalized for ≥24 hours between 1 January 2013 and 31 December 2014 in 23 Dutch hospitals (515 department evaluations).

Main variable

CQI Inpatient Hospital Care questionnaire.

Results

CFA results showed a good fit on individual level (CFI=0.96, TLI=0.95, RMSEA=0.04), which was comparable between specialties. When scores were aggregated to the department level, the fit was less desirable (CFI=0.83, TLI=0.81, RMSEA=0.06), and there was a significant overlap between communication with doctors and explanation of treatment subscales. Departments and hospitals explained ≤5% of total variance in subscale scores. In total, 4-8 departments and 50 respondents per department are needed to reliably evaluate subscales rated on a 4-point scale, and 10 departments with 100-150 respondents per department for binary subscales.

Discussion and conclusions

The CQI Inpatient Hospital Care is a valid and reliable questionnaire to evaluate inpatient experiences in Dutch hospitals provided sufficient sampling is done. Results can facilitate meaningful comparisons and guide quality improvement activities in individual departments and hospitals.

Context

Medical educational reform includes enhancing role modelling of clinical teachers. This requires faculty being aware of their role model status and performance. We developed the System for Evaluation of Teaching Qualities (SETQ) to generate individualized feedback on previously defined teaching qualities and role model status for faculty in (non) academic hospitals.

Objectives

(i) To examine whether teaching qualities of faculty were associated with their being seen as a specialist role model by residents, and (ii) to investigate whether those associations differed across residency years and specialties.

Methods & materials

Cross-sectional questionnaire survey amongst 549 Residents of 36 teaching programs in 15 hospitals in the Netherlands. The main outcome measure was faculty being seen as specialist role models by residents. Statistical analyses included (i) Pearson's correlation coefficients and (ii) multivariable logistic generalized estimating equations to assess the (adjusted) associations between each of five teaching qualities and 'being seen as a role model'.

Results

407 residents completed a total of 4123 evaluations of 662 faculty. All teaching qualities were positively correlated with 'being seen as a role model' with correlation coefficients ranging from 0.49 for 'evaluation of residents' to 0.64 for 'learning climate' (P<0.001). Faculty most likely to be seen as good role models were those rated highly on 'feedback' (odds ratio 2.91, 95% CI: 2.41-3.51), 'a professional attitude towards residents' (OR 2.70, 95% CI: 2.34-3.10) and 'creating a positive learning climate' (OR 2.45, 95% CI: 1.97-3.04). Results did not seem to vary much across residency years. The relative strength of associations between teaching qualities and being seen as a role model were more distinct when comparing specialties.

Conclusions

Good clinical educators are more likely to be seen as specialist role models for most residents.

Background

Medical faculty's teaching performance is often measured using residents' feedback, collected by questionnaires. Researchers extensively studied the psychometric qualities of resulting ratings. However, these studies rarely consider the number of response categories and its consequences for residents' ratings of faculty's teaching performance. We compared the variability of residents' ratings measured by five- and seven-point response scales.

Methods

This retrospective study used teaching performance data from Dutch anaesthesiology residency training programs. Questionnaires with five- and seven-point response scales from the extensively studied System for Evaluation of Teaching Qualities (SETQ) collected the ratings. We inspected ratings' variability by comparing standard deviations, interquartile ranges, and frequency (percentage) distributions. Relevant statistical tests were used to test differences in frequency distributions and teaching performance scores.

Results

We examined 3379 residents' ratings and 480 aggregated faculty scores. Residents used the additional response categories provided by the seven-point scale - especially those differentiating between positive performances. Residents' ratings and aggregated faculty scores showed a more even distribution on the seven-point scale compared to the five-point scale. Also, the seven-point scale showed a smaller ceiling effect. After rescaling, the mean scores and (most) standard deviations of ratings from both scales were comparable.

Conclusions

Ratings from the seven-point scale were more evenly distributed and could potentially yield more nuanced, specific and user-friendly feedback. Still, both scales measured (almost) similar teaching performance outcomes. In teaching performance practice, residents and faculty members should discuss whether response scales fit their preferences and goals.

An optimal learning climate is crucial for the quality of residency training and may also improve residents' well-being and empathy. We investigated the associations of learning climate with residents' work-related well-being. A multicenter questionnaire study was performed among 271 surgery and gynaecology residents in 21 training programs from September 2012 to February 2013. Residents were asked to complete work-related well-being measurements: work engagement (Utrecht Work Engagement Scale), job and specialty satisfaction (measures from Physician Worklife Study), and physician empathy (Jefferson Scale of Physician Empathy). The Dutch Residency Educational Climate Test was used to evaluate learning climate. Multivariate adjusted linear regression analyses were used to estimate associations of learning climate with work-related well-being measures. Well-being measures were completed by 144 residents (53.1%). Learning climate was evaluated by 193 residents, yielding 9.2 evaluations per training program on average. Overall learning climate score was positively associated with work engagement [regression coefficient b = 0.58; 95% confidence interval (CI) 0.18-0.98; p = 0.004] and job satisfaction (b = 0.80; 95% CI 0.48-1.13; p < 0.001). No associations were found between learning climate and empathy and specialty satisfaction. Residents' work engagement and job satisfaction are positively related to the learning climate and may be further enhanced by improved learning climates of training programs.

Background

In fledgling areas of research, evidence supporting causal assumptions is often scarce due to the small number of empirical studies conducted. In many studies it remains unclear what impact explicit and implicit causal assumptions have on the research findings; only the primary assumptions of the researchers are often presented. This is particularly true for research on the effect of faculty's teaching performance on their role modeling. Therefore, there is a need for robust frameworks and methods for transparent formal presentation of the underlying causal assumptions used in assessing the causal effects of teaching performance on role modeling. This study explores the effects of different (plausible) causal assumptions on research outcomes.

Methods

This study revisits a previously published study about the influence of faculty's teaching performance on their role modeling (as teacher-supervisor, physician and person). We drew eight directed acyclic graphs (DAGs) to visually represent different plausible causal relationships between the variables under study. These DAGs were subsequently translated into corresponding statistical models, and regression analyses were performed to estimate the associations between teaching performance and role modeling.

Results

The different causal models were compatible with major differences in the magnitude of the relationship between faculty's teaching performance and their role modeling. Odds ratios for the associations between teaching performance and the three role model types ranged from 31.1 to 73.6 for the teacher-supervisor role, from 3.7 to 15.5 for the physician role, and from 2.8 to 13.8 for the person role.

Conclusions

Different sets of assumptions about causal relationships in role modeling research can be visually depicted using DAGs, which are then used to guide both statistical analysis and interpretation of results. Since study conclusions can be sensitive to different causal assumptions, results should be interpreted in the light of causal assumptions made in each study.

Background

Postgraduate medical education prepares residents for delivery of high quality patient care during training as well as for later practice, which makes high quality residency training programs crucial to safeguard patient care. Healthy learning climates contribute to high quality postgraduate medical education. In several countries, modernization of postgraduate medical education has resulted in hospital-wide responsibilities for monitoring learning climates. This study investigates the association between the actions undertaken by hospital-wide education committees and learning climates in postgraduate medical education.

Methods

Research conducted in December 2010 invited 57 chairs of hospital-wide education committees to complete a questionnaire on their implemented level of quality improvement policies. We merged the survey data from 21 committees that oversaw training programs and used the Dutch Residency Educational Climate Test (D-RECT) instrument in 2012 to measure their training programs' learning climate. We used descriptive statistics and linear mixed models to analyse associations between the functioning of hospital-wide education committees and corresponding learning climates.

Results

In total, 812 resident evaluations for 99 training programs in 21 teaching hospitals were available for analysis. The implementation level of the internal quality management systems as adopted by the hospital-wide education committees varied from 1.6 to 2.6 on a 5 point Likert-scale (ranging from 1 (worst) to 5 (best)). No significant associations were found between the functioning of the committees and corresponding learning climates.

Conclusions

The contribution of hospital-wide committees to creating healthy learning climates is yet to be demonstrated. The absence of such an association could be due to the lack of a Plan-Do-Check-Act cycle guiding the policy as implemented by the committees and the lack of involvement of departmental leadership. Insight into the impact of these strategies on learning climates will benefit the quality of postgraduate medical education and, hopefully, patient care.

AbstractBackgroundMulti-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60–70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change.MethodsThis observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists’ reported change in actual practice.ResultsIn total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p < 0.05) as did negative scores offered by colleagues. (regression coefficient beta = −0.157, p < 0.05). The explained variance of these two variables combined was 34 percent.ConclusionsPerceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors.

Objective

Previous studies identified different typologies of role models (as teacher/supervisor, physician and person) and explored which of faculty's characteristics could distinguish good role models. The aim of this study was to explore how and to which extent clinical faculty's teaching performance influences residents' evaluations of faculty's different role modelling statuses, especially across different specialties.

Methods

In a prospective multicenter multispecialty study of faculty's teaching performance, we used web-based questionnaires to gather empirical data from residents. The main outcome measures were the different typologies of role modelling. The predictors were faculty's overall teaching performance and faculty's teaching performance on specific domains of teaching. The data were analyzed using multilevel regression equations.

Results

In total 219 (69% response rate) residents filled out 2111 questionnaires about 423 (96% response rate) faculty. Faculty's overall teaching performance influenced all role model typologies (OR: from 8.0 to 166.2). For the specific domains of teaching, overall, all three role model typologies were strongly associated with "professional attitude towards residents" (OR: 3.28 for teacher/supervisor, 2.72 for physician and 7.20 for the person role). Further, the teacher/supervisor role was strongly associated with "feedback" and "learning climate" (OR: 3.23 and 2.70). However, the associations of the specific domains of teaching with faculty's role modelling varied widely across specialties.

Conclusion

This study suggests that faculty can substantially enhance their role modelling by improving their teaching performance. The amount of influence that the specific domains of teaching have on role modelling differs across specialties.

Abstract Background Residents are vital to the clinical workforce of today and tomorrow. Although in training to become specialists, they also provide much of the daily patient care. Residency training aims to prepare residents to provide a high quality of care. It is essential to assess the patient outcome aspects of residency training, to evaluate the effect or impact of global investments made in training programs. Therefore, we conducted a systematic review to evaluate the effects of relevant aspects of residency training on patient outcomes. Methods The literature was searched from December 2004 to February 2011 using MEDLINE, Cochrane, Embase and the Education Resources Information Center databases with terms related to residency training and (post) graduate medical education and patient outcomes, including mortality, morbidity, complications, length of stay and patient satisfaction. Included studies evaluated the impact of residency training on patient outcomes. Results Ninety-seven articles were included from 182 full-text articles of the initial 2,001 hits. All studies were of average or good quality and the majority had an observational study design. Ninety-six studies provided insight into the effect of 'the level of experience of residents' on patient outcomes during residency training. Within these studies, the start of the academic year was not without risk (five out of 19 studies), but individual progression of residents (seven studies) as well as progression through residency training (nine out of 10 studies) had a positive effect on patient outcomes. Compared with faculty, residents' care resulted mostly in similar patient outcomes when dedicated supervision and additional operation time were arranged for (34 out of 43 studies). After new, modified or improved training programs, patient outcomes remained unchanged or improved (16 out of 17 studies). Only one study focused on physicians' prior training site when assessing the quality of patient care. In this study, training programs were ranked by complication rates of their graduates, thus linking patient outcomes back to where physicians were trained. Conclusions The majority of studies included in this systematic review drew attention to the fact that patient care appears safe and of equal quality when delivered by residents. A minority of results pointed to some negative patient outcomes from the involvement of residents. Adequate supervision, room for extra operation time, and evaluation of and attention to the individual competence of residents throughout residency training could positively serve patient outcomes. Limited evidence is available on the effect of residency training on later practice. Both qualitative and quantitative research designs are needed to clarify which aspects of residency training best prepare doctors to deliver high quality care.

Abstract Background There is a global need to assess physicians' professional performance in actual clinical practice. Valid and reliable instruments are necessary to support these efforts. This study focuses on the reliability and validity, the influences of some sociodemographic biasing factors, associations between self and other evaluations, and the number of evaluations needed for reliable assessment of a physician based on the three instruments used for the multisource assessment of physicians' professional performance in the Netherlands. Methods This observational validation study of three instruments underlying multisource feedback (MSF) was set in 26 non-academic hospitals in the Netherlands. In total, 146 hospital-based physicians took part in the study. Each physician's professional performance was assessed by peers (physician colleagues), co-workers (including nurses, secretary assistants and other healthcare professionals) and patients. Physicians also completed a self-evaluation. Ratings of 864 peers, 894 co-workers and 1960 patients on MSF were available. We used principal components analysis and methods of classical test theory to evaluate the factor structure, reliability and validity of instruments. We used Pearson's correlation coefficient and linear mixed models to address other objectives. Results The peer, co-worker and patient instruments respectively had six factors, three factors and one factor with high internal consistencies (Cronbach's alpha 0.95 - 0.96). It appeared that only 2 percent of variance in the mean ratings could be attributed to biasing factors. Self-ratings were not correlated with peer, co-worker or patient ratings. However, ratings of peers, co-workers and patients were correlated. Five peer evaluations, five co-worker evaluations and 11 patient evaluations are required to achieve reliable results (reliability coefficient ≥ 0.70). Conclusions The study demonstrated that the three MSF instruments produced reliable and valid data for evaluating physicians' professional performance in the Netherlands. Scores from peers, co-workers and patients were not correlated with self-evaluations. Future research should examine improvement of performance when using MSF.