RT Journal Article
SR Electronic
T1 Changing estimates of leadership ability before a programme: retrospective self-assessments and response-shift bias
JF BMJ Leader
JO BMJ Leader
FD BMJ Publishing Group Ltd
SP 64
OP 67
DO 10.1136/leader-2021-000586
VO 7
IS 1
A1 Oscar Lyons
A1 Harry Kingsley-Smith
A1 Koot Kotze
A1 Karandeep Nandra
A1 Joao R Galante
A1 Nicholas Fahy
A1 Richard Canter
YR 2023
UL http://bmjleader.bmj.com/content/7/1/64.abstract
AB Background Most evaluations of clinical leadership development programmes rely on self-assessments. Self-assessments are vulnerable to response-shift bias. Using retrospective then-tests may help to avoid this bias.In this study, we investigate whether post-programme then-tests (retrospective self-assessments) are more sensitive to change in clinical leadership development programme participants than traditional pre-programme pre-tests when paired with post-test self-assessments.Methods 17 healthcare professionals participated in an 8-month single-centre multidisciplinary leadership development programme. Participants completed prospective pre-test, retrospective then-test and traditional post-test self-assessments using the Primary Colours Questionnaire (PCQ) and Medical Leadership Competency Framework Self-Assessment Tool (MLCFQ). Pre–post pairs and then–post pairs were analysed for changes using Wilcoxon signed-rank tests and compared with a parallel multimethod evaluation organised by Kirkpatrick levels.Results A greater number of significant changes were detected using then-test pairs than pre-test pairs for both the PCQ (11 of 12 vs 4 of 12 items) and MLCFQ (7 of 7 vs 3 of 7 domains). The multimethods data showed positive outcomes at all Kirkpatrick levels.Conclusions In ideal circumstances, both pre-test and then-test evaluations should be conducted. We cautiously suggest that if only one post-programme evaluation can be conducted, then-tests may be appropriate means of detecting change.