Article Text
PDF
Abstract
Significant investments have been made in health research capacity development (HRCD) initiatives focusing on individual professionals and institutional frameworks. However, prevailing approaches often prioritise specific research projects over explicit strategies for strengthening the capacities of health research professionals (HRPs) particularly in low and middle-income countries. Despite recognition of its limitations, this implicit approach persists resulting in a lack of effective HRCD strategies. Additionally, the absence of globally standardised mechanisms for evaluating HRCD initiatives makes it more complex to define clear success benchmarks for these initiatives. Evaluations of HRCD strategies predominantly focus on pre-intervention and post-intervention assessments of specific interventions often neglecting the broader context of capacity development. Consequently, there is an imperative for a more systematic approach to measuring HRCD, particularly at the individual level.
This paper describes the design and the scope of the Professional Development Scheme (PDS), a tool designed to quantify HRCD among HRPs. Structured across four core sections including professional experiences, qualifications and the self-evaluation of 325 key competencies, the PDS allows users to generate evidence of existing strengths and possible areas of improvement in their research skills. Profile submissions undergo rigorous moderation to ensure fidelity and uniformity in competency evaluation.
The PDS can offer a structured approach to assessing and improving research capacities among HRPs. By focusing on specific skills and employing clear evaluation methods, the PDS aims to overcome the shortcomings of previous approaches and promote development in global health research capacity.
- capacity
- career development
- competencies
- development
- assessment
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Considerable economic and social efforts have been invested in health research capacity development (HRCD) initiatives targeted at individual and institutional levels.1 2 This investment in capacity development has predominantly been based on the assumption that research capacities of health research professionals (HRPs) will be developed through their involvement in the research activities.3 This is particularly evident for HRPs working in low and middle-income countries (LMICs). This ‘implicit’ approach to capacity development4 is known to be largely ineffective, yet it is regularly adopted.5–7 The lack of globally recognised mechanisms to evaluate HRCD initiatives makes it difficult to establish clear benchmarks for their success.8–11 Most of the work done to evaluate HRCD strategies focuses on pre-evaluation and post-evaluation of a specific HRCD intervention12–14 aiming at capitalising on the lessons learned from existing good practices.
The tools used in these evaluations primarily focus on a specific step of the complex health research process and do not measure the overall HRCD.12 This missing evidence does not allow the iterative process of improving HRCD initiatives, limiting their effectiveness. It also prevents research institutions in LMICs from assuming leading roles in health research globally.2
An important step to address this gap is to develop a tool that measures globally applicable, quantifiable parameters of HRCD at an individual level. Systematically measuring HRCD by evaluating personal competencies could effectively provide evidence of the existing research knowledge in LMICs and support the design of structured training plans that address gaps in research competencies.15 This could ultimately support research activities that place ‘explicit’ priority on HRCD.2
Aims
This paper describes the Professional Development Scheme (PDS), a tool developed to measure HRCD in HRPs based in LMICs including its development and key features.
The tool can be helpful for HRPs at an individual level and support the work of healthcare leaders at team, institutional and partnership levels to assess the competencies of the group of researchers they are asked to manage and coordinate.
The professional development scheme
The origins of the PDS
The Global Health Network and TDR Global competency framework
The PDS is built on a comprehensive study that determined the set of competencies required to approach, run and deliver clinical research identified in 2014 by The Global Health Network (TGHN) of the University of Oxford and The Special Programme for Research and Training in Tropical Diseases co-sponsored by the UNICEF, the United Nations Development Programme, the World Bank and the WHO when they created the Global Competency Framework for Clinical Research (the Framework).15
The Framework combined 28 existing frameworks listing research competencies with information from 116 job descriptions obtained from clinical trial units worldwide and specialised online resources.16 The initial data set used to create the Framework focused on the research competencies required to run clinical trials. However, since a similar set of competencies is required across different research designs, the scope of the framework design exercise was expanded to obtain a framework applicable to clinical trials as well as a wide range of other studies that may differ in design, geographical location or disease but can be adapted to the particular needs of specific projects or roles.
The resulting framework was refined using stakeholders’ feedback. It included a set of 50 key competencies required throughout the research study lifecycle, from scientific literature assessment to results dissemination via project management, public engagement or grant application. The Framework was complemented by a Competencies Dictionary where details on each of the 50 competencies were defined and further clarified using keywords. The dictionary also lists the abilities required to master each competency. The abilities identify the practical know-how needed to perform the competency.
From abilities to professional competencies
The abilities listed and described in the dictionary under each of the 50 competencies identified in the Framework were the basis for creating the PDS.17
A team of experts selected from the TGHN network used the abilities included in the Competency Dictionary to generate a list of statements for each competency. These statements represent the core section of the PDS against which its users are asked to measure their research knowledge. This was an iterative process. A more extensive list of competencies was created and included in the new version of the PDS in 2018. In this new version, 203 new competencies were added to the original 122 taking the PDS competencies list to 325.
The PDS structure
The PDS online tool is accessible free of charge from the TGHN website through a basic login registration process.17 The PDS aims to provide an extensive mechanism for capturing core competencies, qualifications and training at every level of experience from a recent graduate in their first post to a director managing a large research centre.
Members of the PDS community of practice (from now on: ‘the members’) are asked to create their PDS profile, entering information on their professional experience and qualifications in the four sections of the PDS online tool. The average time required to complete the PDS in all its parts varies between 35 and 55 min. The four sections of the PDS are: ‘Research Roles, Qualifications & Location’, ‘Research Competencies’, ‘Further Professional Development’ and ‘Create your CV’ (figure 1).
The four sections that compose the PDS online tool as they appear to members when they access the PDS webpage. They guide the members in collecting the information needed to access their research competencies. CV, curriculum vitae; PDS, Professional Development Scheme.
Acting as a unified training log, the four PDS sections facilitate recording all competencies needed to run high-quality research including soft skills acquired through informal skills-learning processes such as timekeeping and communication skills. The data collected in the four sections are essential to provide information for the moderation system that supports the PDS.
Each section is identifiable within the PDS webpage18 (availabele as online supplemental file 2) rand a brief explanation of the section’s aims and requirements is given together with an estimated length of time needed for completion. To facilitate the use of the PDS in low bandwidth settings, the PDS offers in each section and on each page the possibility of saving and completing the inclusion of information at a later time point. This also facilitates the completion of the longer PDS sections in subsequent steps.
Supplemental material
‘Research Roles, Qualifications & Location’
This first section collects preliminary information on the member’s academic qualifications and professional experience. A series of dropdown menus using standard definitions to avoid mismatched classification ask the member to include information on their current geographical location, present and previous research roles and academic qualifications. The complete list of included job roles and academic qualifications is presented in online supplemental appendix 1.
Supplemental material
‘Research Competencies’
In this section, PDS members are asked to self-evaluate their knowledge to perform the 325 competencies derived from the Framework as previously described. The section is divided into five categories of competencies: ‘Professional Skills’, ‘Research Operations’, ‘Ethics, Quality & Risk Management’, ‘Study & Site(s) Management’ and ‘Scientific Thinking’.
The complete list of research competencies, divided into subcategories and groups of competencies, that the members of the PDS are asked to self-assess is available as supplementary material (online supplemental appendix 2).
The division of the ‘Research Competencies’ section into categories, subcategories and groups aims to facilitate the members’ understanding of the specific context of the competency they are asked to self-score. An example of the PDS ‘Research competencies’ webpage layout is given in online supplemental appendix 3. These five categories are also colour-coded on the webpage, offering visual support in filling it out appropriately.
Each member is asked to self-score all 325 competencies, regardless of their professional role or the relevance of a category of competencies to their daily job, to detect 'hidden' competencies which are learnt mainly by performing a task or being exposed to working in a team with other professionals.
The members score their knowledge around each of the competencies by applying a specific 6-point Likert scale adapted to the requirements of the PDS as per supplementary material (online supplemental appendix 4).
Once self-assessment of the 325 competencies in the PDS is completed, the member can access their competency wheel (figure 2).
The Professional Development Scheme Competency Wheel visually represents strengths and gaps in the member’s research competencies. The example given below clearly presents how the selected member has higher competencies in the ‘Professional Skills’ and ‘Study and Site(s) Management’ categories.
The competency wheel effectively visualises strengths and gaps in the member’s research competencies. It incorporates the median score self-assigned in each category and subcategory and presents it as a spider diagram. The visual representation may foster members’ confidence in using the competencies acquired and provide indications of gaps.
‘Further Professional Development’
Each member is asked to include additional details on professional and personal experiences. The information collected here may not be directly related to the professional qualifications described in the section ‘Research Roles, Qualifications & Location’ but is relevant to determining a member’s degree of competency. This section recognises that organisational, leadership, networking, presentation and dissemination skills can be acquired through activities that differ from those linked to obtaining professional or academic qualifications.
This section works through a series of dropdown menus requiring the user to select the professional development activity performed and the action taken to execute it. The types of professional activities and subsequent actions that constitute the content of the two dropdown menus mentioned above are listed in online supplemental appendix 5.
The members are then invited to attach supporting documentation. This serves the double purpose of confirming the appropriateness of the information entered and creating a single repository of all documentation supporting further professional development activities undertaken by the member.
‘Create your CV’
The final section of the PDS gives the chance to use the information entered in previous sections to create a professional curriculum vitae (CV).
The generated CV conforms to Good Clinical Practice recommendations and automatically includes the courses that the member has taken using the TGHN training platform. Members may attach their competency wheel to their CVs as a visual representation of their research competencies.
The PDS scoring system
The primary goal of the PDS is to numerically quantify each member’s research competencies. A scoring system has been created for this purpose. Each member’s PDS final score is determined by the sum of subscores obtained by completing the different sections of their PDS profile.
The baseline score
The information entered in the ‘Research Roles, Qualifications & Location’ section generates the first set of points. The member’s specific score in this section depends on academic qualifications, job roles and the number of years they spent in each. The matrix in online supplemental appendix 1 gives a clear overview of these scores.
Self-evaluation scoring exercise
As mentioned in the ‘Research Competencies’ section, PDS members are invited to self-evaluate their research competencies by quantifying their level of knowledge and ability to perform each of the specific 325 competencies listed in the PDS through the use of a 6-point Likert scale.
A set of points ranging from 0 to 22 has been assigned to each point on the Likert scale (table 1).
Points lined to the 6-point Likert scale used to self-assess research competencies
The choice of the 0–22 scale was based on the need to give adequate relevance to the self-evaluation scoring exercise compared with points acquired in other sections of the PDS.
The competencies weighting system
To increase the accuracy of the score generated in the self-scoring exercise, each of the 325 competencies was assigned a weighted score according to the level of complexity and expertise needed to perform the task linked to that competency using a multiplicator factor scaling between 0 and 5. For example, the competency ‘Interprets and summarises complex issues around study or disease area’ was weighted 4 since the skills linked to this competency are more complex than the one required to ‘identify errors and inaccuracies within research documents and data sets’, which is weighted as 1.
The number of points assigned to a member for each of the 325 competencies results from the points linked to the self-assigned score in the Likert scale (0–22) multiplied by the weight of each question (0–5). The final score obtained in the ‘Research Competencies’ section of the PDS is, therefore, the sum of the scores derived by the member’s ability to perform each competency-related activity and the complexity of that activity. For example, by scoring 5 on the Likert scale in the competency ‘Interprets and summarises complex issues around study or disease area’, which is weighted 4, the total number of points that competency will contribute to the final score will be 88; 4 (the weight of the question) multiplied by 22 (the points assigned to scoring 5 in the Likert scale used in the self-scoring system). In the case of ‘identify errors and inaccuracies within research documents and data sets’, the total number of points added to the total score if the member assigns a 4 on the Likert scale will be 18, as the result of 1 (the weight of the competency) multiplied 18 (the points assigned to scoring 4 in the Likert scale used in the self-scoring system).
Score associated with additional professional activities
Finally, additional points are assigned to a member in relation to further professional achievements or certifications. Information on these activities is gathered in the ‘Further Professional Development’ PDS section. The complete list of the activities that can be selected in this section and the related score assigned can be found in online supplemental appendix 5.
More complex activities such as ‘Led a conference’ score a proportionately weighted higher number of points (n=100) than ‘Attended a conference’ (n=10) ‘Presented at a conference’ (n=40) or ‘Organised a conference’ (n=50).
The scores assigned in this section measure the proactivity of the PDS member in seeking networking activities, engagement with the scientific community they are part of and proactively looking for professional development opportunities.
The PDS moderation process
On completing the PDS profile, members submit their profile for moderation. The moderation process was designed to increase the reliability and robustness of the PDS as a tool to measure research capacities and monitor personal progression. As previously detailed, the PDS score is obtained by points derived from qualifications, training and experience.
The moderation process—performed by an international team of qualified moderators—checks the coherence between the scores self-assigned in each competency and the qualifications and experience the members declared in the relevant sections of the PDS.
The team of moderators changes over time due to the voluntary basis of the role. It generally includes between three and five professionals from different healthcare professions involved in research.
A predictive algorithm that could automatically moderate users’ competency scores was created to streamline the moderation process. The codebase and algorithm associated with the automatic moderation system are based on the principle of machine learning19 to improve efficiency.
Usually, around 15% of submitted profiles do not pass the auto-moderation process and are raised for manual review. The profile is assigned to a member of the moderators’ team who performs the moderation using a validated moderation manual to ensure accuracy and consistency.
When the profile has been moderated, automatically or manually, the member receives an autogenerated email stating the overall score and membership level. A membership certificate is also available to download from their profile.
The levels of membership
Professional membership scheme levels have been created to support the understanding of the assigned score. The PDS members are assigned a PDS Professional Membership Scheme Level based on the score assigned. There are five levels: Foundation, affiliate, professional, associate and fellow. Each level is then divided into five sublevels to record the progressive acquisition of competencies even when the score variation remains within the range assigned to one of the five levels.
PDS for research teams
In 2021, a new feature was introduced to the PDS that allows the profiles of individuals working together to be presented and interrogated at different levels within a team profile. It enables the assessment of research competencies across a research team, study programme or consortia and organisation, helping individuals and organisational leadership to consider the impact of professional development and capability-building initiatives on the competency of individuals and teams. The team dashboard provides three main visualisations of the team’s research competencies described below.
High-level snapshot across teams
This visualisation offers an overview of the team’s capacities with the aim of assessing skill gaps and which team members should be assigned to additional training to fill in these gaps. High-scoring and low-scoring members by percentiles are also shown. Further breakdown by individual competency provides quick views into top and bottom scoring members, allows for rapid analysis of team strengths and can be used as an indicator of training needs.
Time-series across teams
This second visualisation aims to assess consistent skill gaps over time as well as the overall team balance in terms of median competency scores over time. Data sets can also be downloaded for further analyses and the breakdown data via competency area is shown.
A visual representation of the overall team dashboard and overtime skills dashboard can be found in online supplemental appendix 6.
Individual team member view
This third visualisation aims to enable the team to review the impact of a study on the development of the team and of individual team members, both at a defined time point and over time. This view allows more in-depth views at the individual level. Team members can be managed as well as new members invited via the site manager/administrator.
The global uptake of the PDS as a tool to measure HRCD
The PDS has demonstrated substantial global engagement with over 75 000 page views in 2023 alone, attracting users from 153 countries. Since its inception in 2010, the PDS has consistently risen in profiles, reaching 16 382 by the end of 2023. Although there was a slight dip in new profile creations from 2021 to 2022 (2372 users—2021 and 1722 users—2022) attributed in part to the pandemic’s impact on front-line health staff availability, the scheme continues to draw interest and participation and there was a rise again in 2023 (2025 users).
Geographically, the global distribution of PDS members in 2023 showcased diversity with significant representation from countries such as Kenya, India and Nigeria. Notably, outreach and promotion efforts by study groups and TGHN collaborators in these countries correlated with increased participation. The scheme has particularly resonated with users from LMICs: Kenya, Nigeria, India, Uganda and Cameroon are the top five countries by user numbers since the scheme’s inception. To 2023, 55% of users are from Africa, 20% from Asia, 13% from Europe, 7% from North America, 4% from Latin America and the Caribbean and 2% from Oceania.
It should be noted that the PDS is currently available online only in English. Translations in Thai, Portuguese and Spanish are available offline but the developers are working to integrate these and other languages into the online PDS system.
To enhance the scheme’s effectiveness, TGHN introduced the PDS Dashboard in 2021, providing a comprehensive view across teams. As the team dashboard continues to roll out, it is anticipated to enhance further the professional development and training initiatives within TGHN.
The PDS has been promoted among the users of the various knowledge exchange hubs managed by TGHN. The number of TGHN thematic hubs and related uses is increasing and the number of PDS members is expected to grow at the same speed as in previous years.
A specific focus will be given to improving the use of the PDS in Latin America and the Caribbean. This reflects TGHN’s effort to promote its contents in Spanish and Portuguese which included the offline translation of the PDS in those languages.
Conclusion
The PDS aims to support HRPs in identifying the level of their ability to perform clinical research activities. It is a useful tool for quantitatively identifying competencies gaps in individuals and teams and may represent valid support to healthcare leaders at team, institutional and partnership levels in assessing the competencies of the group of researchers they are asked to manage and coordinate.
Using the PDS to measure the change in research competencies over time provides a mechanism for assessing the impact of HRCD initiatives.
The PDS can facilitate the identification of competency gaps for users and their institutions. This may be a valuable support to prioritise training needs, providing access to relevant courses and training opportunities. In an iterative process, the PDS can map this acquisition of capacities through updating the relevant section of the PDS profile.
Similarly, the PDS can be used to measure the impact of ‘learning by doing’ due to the work performed on a particular study. Measuring and providing evidence to the process of competency acquisition represents the critical element in making HRCD an ‘explicit’ outcome of research activities.
At a different level, the PDS provides an opportunity to rebalance the power dynamics within research teams, supporting the creation of career development plans for more technical and administrative roles, still relatively unsupported in their professional growth.7 20
Furthermore, the PDS can be a valuable tool for early career researchers and particularly for researchers working in the Global South. Because they are generally less exposed to highly recognised training opportunities, they can benefit from the evidence of training needs generated by the PDS when defining their career paths.
In conclusion, the PDS might benefit from future research focusing on a systematic evaluation of the strengths and weaknesses of its real-world use. This is particularly relevant considering that the Global Health space has evolved towards the inclusion of a plurality of approaches to health that may need to be incorporated in the technical measurement that the PDS aims to provide. Asking what competencies the PDS is measuring and who may benefit from its use may provide valuable insight to support the streamlining of the goals that this versatile tool can help achieve.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Acknowledgments
We express our gratitude to colleagues at The Global Health Network and TDR who supported the creation of the PDS from its original idea to the moment it went online as well as to the very qualified researchers who gave input throughout the tool's revision process. A special mention goes to all PDS users who contributed to this work with comments, improvement ideas and suggestions.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
X @BilardiDavide
Contributors DB and ER created and circulated the first concept with the other authors. DB, ER, KS, SZ, BLB and TL critically reviewed the manuscript and substantially contributed to its various versions. DB implemented the various contributions into the various versions of the manuscript. ER and TL revised the advance draft for further revision and corrections. DB finalised the document. All authors approved the final version of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.