PT - JOURNAL ARTICLE AU - Oscar Lyons AU - Harry Kingsley-Smith AU - Koot Kotze AU - Karandeep Nandra AU - Joao R Galante AU - Nicholas Fahy AU - Richard Canter TI - Changing estimates of leadership ability before a programme: retrospective self-assessments and response-shift bias AID - 10.1136/leader-2021-000586 DP - 2023 Mar 01 TA - BMJ Leader PG - 64--67 VI - 7 IP - 1 4099 - http://bmjleader.bmj.com/content/7/1/64.short 4100 - http://bmjleader.bmj.com/content/7/1/64.full SO - BMJ Leader2023 Mar 01; 7 AB - Background Most evaluations of clinical leadership development programmes rely on self-assessments. Self-assessments are vulnerable to response-shift bias. Using retrospective then-tests may help to avoid this bias.In this study, we investigate whether post-programme then-tests (retrospective self-assessments) are more sensitive to change in clinical leadership development programme participants than traditional pre-programme pre-tests when paired with post-test self-assessments.Methods 17 healthcare professionals participated in an 8-month single-centre multidisciplinary leadership development programme. Participants completed prospective pre-test, retrospective then-test and traditional post-test self-assessments using the Primary Colours Questionnaire (PCQ) and Medical Leadership Competency Framework Self-Assessment Tool (MLCFQ). Pre–post pairs and then–post pairs were analysed for changes using Wilcoxon signed-rank tests and compared with a parallel multimethod evaluation organised by Kirkpatrick levels.Results A greater number of significant changes were detected using then-test pairs than pre-test pairs for both the PCQ (11 of 12 vs 4 of 12 items) and MLCFQ (7 of 7 vs 3 of 7 domains). The multimethods data showed positive outcomes at all Kirkpatrick levels.Conclusions In ideal circumstances, both pre-test and then-test evaluations should be conducted. We cautiously suggest that if only one post-programme evaluation can be conducted, then-tests may be appropriate means of detecting change.