You are here

Article Text

Article menu
Download PDFPDF
Brief report
Changing estimates of leadership ability before a programme: retrospective self-assessments and response-shift bias
  1. Oscar Lyons1,2,
  2. Harry Kingsley-Smith3,
  3. Koot Kotze2,
  4. Karandeep Nandra3,4,
  5. Joao R Galante3,5,
  6. Nicholas Fahy2,6,
  7. Richard Canter1
  1. 1 Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
  2. 2 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  3. 3 Department of Medical Education, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
  4. 4 Public Health, East of England Deanery, Cambridge, UK
  5. 5 Kent Oncology Centre, Maidstone and Tunbridge Wells NHS Trust, Maidstone, UK
  6. 6 Health and Wellbeing Research Group, RAND Europe, Cambridge, UK
  1. Correspondence to Dr Oscar Lyons, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX2 6GG, UK; oscar.lyons{at}phc.ox.ac.uk

Abstract

Background Most evaluations of clinical leadership development programmes rely on self-assessments. Self-assessments are vulnerable to response-shift bias. Using retrospective then-tests may help to avoid this bias.

In this study, we investigate whether post-programme then-tests (retrospective self-assessments) are more sensitive to change in clinical leadership development programme participants than traditional pre-programme pre-tests when paired with post-test self-assessments.

Methods 17 healthcare professionals participated in an 8-month single-centre multidisciplinary leadership development programme. Participants completed prospective pre-test, retrospective then-test and traditional post-test self-assessments using the Primary Colours Questionnaire (PCQ) and Medical Leadership Competency Framework Self-Assessment Tool (MLCFQ). Pre–post pairs and then–post pairs were analysed for changes using Wilcoxon signed-rank tests and compared with a parallel multimethod evaluation organised by Kirkpatrick levels.

Results A greater number of significant changes were detected using then-test pairs than pre-test pairs for both the PCQ (11 of 12 vs 4 of 12 items) and MLCFQ (7 of 7 vs 3 of 7 domains). The multimethods data showed positive outcomes at all Kirkpatrick levels.

Conclusions In ideal circumstances, both pre-test and then-test evaluations should be conducted. We cautiously suggest that if only one post-programme evaluation can be conducted, then-tests may be appropriate means of detecting change.

  • clinical leadership
  • competencies
  • research
  • measurement
  • leadership assessment

https://doi.org/10.1136/leader-2021-000586

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

View Full Text

Footnotes

  • Twitter @oscarlyonsnz

  • Contributors OL, RC, KN and JRG designed the work and acquired the data. All authors interpreted the data, drafted the manuscript and revised it critically for intellectual content. All authors approved the final version of the manuscript. All authors agreed to be accountable for all aspects of the work, ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

  • Funding The Oxford University Hospitals Foundation Trust’s Department of Medical Education kindly provided rooms and catering for the workshops. OL was supported during this work by a Rhodes Scholarship, a Goodger and Schorstein Research Scholarship (University of Oxford) and the Shirtcliffe Fellowship (Universities New Zealand).

  • Competing interests OL, KN, JRG and RC contributed to the programme as faculty. KN and JRG are graduates of the first Emerging Leaders Programme.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.