Developing and Validating Entrustable Professional Activities (EPAs) for Rheumatology Fellowship Training Programs in Saudi Arabia: A Delphi Study

Introduction

The journey of becoming a practicing rheumatologist in Saudi Arabia has undergone significant change in recent decades, notably toward competence-based medical education (CBME). CBME is defined as an evidence-based approach to preparing physicians for practice who possess the desired knowledge, skills and attitudes outlined by the careful consideration of societal and patients’ needs.¹ This approach has been widely adopted worldwide in undergraduate and postgraduate medical education programs to ensure safe medical practitioners who are ready to respond to complex situations.

There are several frameworks for CBME, such as the Canadian Medical Education Directives for Specialists (CanMEDS),² the Accreditation Council for Graduate Medical Education (ACGME),³ the Saudi Medical Education Directives (Saudi MEDs),⁴ and the General Medical Council (GMC).⁵ The Saudi Commission for Health Specialties (SCFHS) adopted the CanMEDS framework to establish a core curriculum for all training programs including rheumatology. This competency-based framework is used to describe the core knowledge, skills, and attitude of graduating rheumatologists.^6–8 Under the SCFHS rheumatology curriculum, Saudi physicians undergo two years of full-time supervised training in which they are actively involved in patient care, with responsibility increasing as further experience and competence are gained. The implementation of these frameworks’ competencies might be challenging to educators, as these competencies are usually describing general characteristics of individuals (physicians) rather than clear and specific tasks or responsibilities.⁹ The concept of entrustable professional activities (EPAs) was introduced in 2005¹⁰ to fill this gap. EPAs are defined as responsibilities or tasks to be entrusted to the unsupervised execution by a trainee once he or she has attained sufficient specific competencies.¹¹

There is an important concept to assure proper application of EPAs in real life. They should not be considered alternatives to competencies, as they are designed to translate competencies into day-to-day clinical practice. In more clearer terms, competencies should describe the physician while the EPAs should describe the work.¹² Each EPA from this perspective requires multiple competencies to utilize. Consequently, assessment of an EPA results in a more holistic and systemic assessment of competencies.¹³

Several published EPAs exist for rheumatology postgraduate training programs, such as the Canadian¹⁴ and the American EPAs.¹⁵ However, it is difficult to adopt a set of EPAs from one culture to another because many factors may influence this process, such as local culture, patients’ values, and the prevailing healthcare system. Institutions around the world that have adopted EPAs in their undergraduate and postgraduate programs have further developed and validated their own EPAs based on their cultural values, their healthcare system and patients’ needs.¹⁶

In Saudi Arabia, efforts are underway to develop and validate EPAs at the internship level.⁴ Otherwise, we are lacking validated EPAs to define the core tasks which graduating residents in different internal medicine specialties must be able to perform independently in practice. This study on rheumatology fellowship EPAs is part of a larger project that aims to develop and validate EPAs in different internal medicine specialties in Saudi Arabia. This work should clearly define tasks and responsibilities in all specialties to improve quality of training, validity of assessment and hence patients care.

Methods

To develop and validate a consensus on end-of-training EPAs for rheumatology fellowship training programs in Saudi Arabia, we conducted a modified Delphi technique–based study that implemented two phases: a preliminary phase involving a meeting of experts and then two rounds of modified Delphi technique. This study is approved by the institutional review board at Umm Al-Qura University, Makkah, Saudi Arabia. Figure 1 represents the design and flow of the current study.

Figure 1 Represent the design and flow of the current study.

The study population is rheumatology experts in Saudi Arabia. This includes those with more than five years of experience in rheumatology and who were participating in postgraduate training programs in rheumatology in Saudi Arabia. The sampling technique was purposeful sampling. Thirty-two experts were nominated and invited to participate in the two-round Delphi study. Written informed consent was obtained from all participants in this study. A brief orientation presentation was sent by Email to develop orientation of the study aim and the nature of EPA development.

The Preliminary phase: a group of four expert rheumatologists, including some of the authors, conducted a comprehensive literature review of international EPAs and CBME rheumatology frameworks, including the SCFHS competency training framework for rheumatology.⁶ They outlined a list of potential end-of-training EPAs for rheumatology fellowship training programs in Saudi Arabia. This was piloted among a group of five rheumatologists and some changes were applied in wordings and rephasing. An initial set of potential end-of-training EPAs in rheumatology marked the end of this phase, which was considered the base of the first-round Delphi online survey.

The first Delphi round: a group of rheumatology experts in Saudi Arabia were nominated and invited to participate in this study. Experts were defined as those who have more than five years of experience in the field of rheumatology and were participating in postgraduate training programs in rheumatology in Saudi Arabia. The researchers tried to invite experts who would represent different programs and affiliations as well as most regions of Saudi Arabia.

The first Delphi round consisted of the preliminary core rheumatology fellowship EPAs. The experts’ panelists were asked to determine the content representativeness and relevance (ie, content validity) of each end-of-training EPA proposed in the preliminary phase based on a 5-point Likert scale ranging from 1 (not important/relevant) to 5 (very important/relevant). In addition to rating the EPAs, participants had the opportunity to comment on wording, make modifications and add new EPAs to the initial set. In addition, participants’ demographics data were collected in the survey including the experiences and affiliations.

The second Delphi round: based on the results of the first Delphi-round, including the developed consensus and comments of the panel, the second-Delphi round was developed. The same experts panel were reinvited to review the list of EAPs and define the representativeness and relevance of each EPAs. The aim of the second round was to verify the experts’ decisions based on the developed consensus from the first round, again using the 5-point Likert scale. There was also a free space to comment on and modify the pre-final EPAs. We asked the panelists about their agreement level on the changes they made, which they could ascertain in the new pre-final set of EPAs. We analyzed the ratings in the second round, thoroughly reviewed the comments and modifications of the panelists, and finally produced the final set of end-of-trainings EPAs in rheumatology.

Statistical Analysis

Descriptive statistics were utilized to describe participants’ demographic characteristics and group responses to each statement in all rounds. A response rate of 80% was considered, accordingly the minimum number of experts to be recruited for the panel was 25 to 30.^11,17 Based on previous literature,^16,18,19 the consensus criteria to retain EPAs from the draft list were set as (1) average of 4 points or higher on the 5-point Likert scale and (2) more than 80% of the panel members rating it as 4 or 5. We used Cronbach’s alpha to measure the internal consistency of the study; the standard deviation (SD) was also used to identify the consensus of the mean values obtained for each part.

Results

Demographic Characteristics of the Rheumatology Experts

We invited 32 experts in rheumatology to participate in first and second rounds of the modified Delphi technique survey. Twenty-five out of those 32 experts (78.12%) completed the first round, while 30 experts completed the second round (93.75%). They were mostly male (80%) with an average age between 45 and 54 years, and they came from different regions and programs in Saudi Arabia. All study participants were primarily involved in the education for fellowship programs in rheumatology. Most of them had more than 10 years of experience as a rheumatologist (84%) and also more than 10 years of experience in the rheumatology fellowship training programs (44%), as shown in Table 1. We have included among our experts the current and former heads of rheumatology fellowship training programs in Saudi Arabia.

Table 1 Demographic Characteristics of the Participants in the First Round of Modified Delphi Technique

The First Round of the Modified Delphi Survey

In the preliminary phase, four rheumatologists developed and piloted an initial set of 36 end-of-training EPAs for rheumatology fellowship training programs in Saudi Arabia (Table 2). These EPAs were categorized on four domains (clinical assessment, clinical management, procedures and transferable skills), and each domain consisted of a list of EPAs. The rheumatology experts were invited to determine the content validity of each EPA. In the first round, most of the 36 EPAs were rated as 4 or 5 by more than 80% of the panel members. Five EPAs were eliminated because a consensus among the experts was not reached. Based on the experts’ comments, one EPA was added (EPA B9 in final Table 2), and modifications were made in two other EPAs. A pre-final set of 32 different EPAs was produced after this round.

Table 2 The Suggested and Modified EPAs in the Two Rounds of Delphi Technique Surveys

The Second Round of the Modified Delphi Survey

These new 32 pre-final EPAs were sent in the second round of the modified Delphi to the same experts for validation using the same rating scale. We asked the panelists about their agreement level on the modifications they made. As the first-round, statements reaching average mean of 4 or higher on the 5-point Likert scale and had more than 80% of agreement as core EPAs were considered a consensus. The mean of all suggested pre-final EPAs of the second round reached an average of 4 or higher and more than 80% agreement, including the newly added and modified EPAs. In addition, the panelists were asked about their agreement of the applied modifications on the pre-final EPAs list based on the group consensus in the first Delphi-round. Seventeen responses agreed with the modifications (57%), six did not agree (20%) and seven were not sure (23%). The study generated a consensus of the final set of end-of-training EPAs in rheumatology fellowship training programs in Saudi Arabia as shown in Table 3.

Table 3 The EPAs for Rheumatology Fellowship Training Program in Saudi Arabia

The internal consistency of the two modified Delphi rounds was measured by Cronbach’s alpha. The values for Cronbach’s alpha for the first and second rounds were α = 0.958 and 0.9229, respectively.

Discussion

Based on a strict methodology, we have developed a list of EPAs for rheumatology fellowship training programs in Saudi Arabia. A core group of Saudi experts’ rheumatologists created an initial set of EPAs based on an extensive literature review and studying the Saudi Rheumatology Fellowship Curriculum⁶ produced by the SCFHS. We did not observe one week of practice to help design the EPAs, as had been suggested once.¹⁷ We also did not work on standard textbooks, as some other research work did.²⁰ The initial approach we used was also has been used by several other research groups.^{16,18,21–23} We used a two-round content survey in the modified Delphi technique to reach a consensus on our initial list of EPAs. This is a common methodology suggested¹⁷ and used in many similar research projects. Our initial number of EPAs and total number of experts involved in the Delphi were decided according to general recommendations by experts in this field.^11,17 We did not create a lengthy list of EPAs—one research group used 285²⁰—and we followed the available guides and tools for creating an effective EPA. We tended to use vague titles for the EPAs, as we left specifications of each EPA to the next level of mapping and identifying milestones for these EPAs.¹¹ We dealt carefully with the comments of our experts and made necessary adjustments in our final list of EPAs entering the second round or in its final shape. The result of our work was a total of 32 EPAs for the rheumatology fellowship training programs in Saudi Arabia.

EPAs define the core tasks that a graduating rheumatologist must be able to perform independently in practice. Our aim in conducting this study was clearly as stated¹¹ to operationalize CBME through a stepwise and safe engagement of trainees in clinical practice. This work should be considered as a framework for workplace curriculum development.¹⁷ Detailed mapping of this framework is the topic of the next rubric to shape the face of the activities during the specified period of training. It is obvious that each EPA requires multiple competencies to perform;²⁴ from this perspective each trainee may take different approaches and variable times to fulfill each EPA. The level of supervision then may change accordingly during the training period. This should also be accompanied by a gradual increase of autonomy among trainees. This is an essential concept that should be clear for both trainees and trainers. Therefore, assessment of an EPA leads to a more holistic assessment of competency.²⁴ Providing such a detailed framework of EPAs of rheumatology in Saudi Arabia, as outlined in this research work, should help clearly define the multiple tools in the assessment process utilizing portfolios and other tools. EPA-based assessment should link educational objectives directly to health care and patient-safety objectives.^17,25

While this study focused on developing EPAs for traditional face-to-face training, the increasing role of online education in rheumatology cannot be overlooked. Online platforms offer several advantages, including flexibility, accessibility, and cost-effectiveness. They can provide learners with opportunities for continuous professional development, regardless of their geographic location or schedule. However, it is essential to recognize the limitations of online education. Face-to-face interactions can be crucial for developing clinical skills, building relationships with mentors, and fostering a sense of community among healthcare professionals. A hybrid approach, combining both online and in-person learning, may be optimal for rheumatology training.

A few considerations should be addressed to achieve beneficial applications of this framework of EPAs in clinical practice. Faculty development in the principles of curriculum development in the workplace utilizing EPAs is a key factor for success. An entrustment-based discussion (EBD) is a structured conversation with a learner to support an entrustment decision for an EPA.¹¹ This should assure a good level of alignment between planned, implemented and hidden curricula. Another important concept is that EPAs do not define all the knowledge, skills and attitude that must be developed during training.²⁰ Clearly, professionals are undertaking activities that are not EPAs.¹⁷ From this perspective, EPAs in this research work should follow the dynamic properties of curriculum development processes. Training programs should have regular reviews for any stated EPAs and should follow a consensus methodology to adjust existing EPAs and/or adopt new ones. The suggested EPAs in this research work are not mandatory, and clearly, they are not definitive in a rapidly growing field like rheumatology.

Composing and conducting research was not an EPA that we designed initially. We did not find a specific one about conducting research in the Canadian EPAs.^2,14 The Saudi Rheumatology Fellowship Curriculum published in 2019⁶ stated the objective for (research rotation) is to “demonstrate basic knowledge” in different competencies related to research principles: creating questions, study designs, different designs of clinical trials, etc. Clearly, the focus is on applying research findings rather than attaining specific skills to compose and conduct actual research. This is not to underestimate the value of conducting research; it is known that not all clinicians are competent in conducting research, but all of them should demonstrate all competencies related to critiquing research findings and applying evidence-based medicine. This was included in our original EPAs (EPA No. D8) and was rated highly among our experts. In addition, composing and conducting research can be a milestone under the umbrella of promoting health of the society (EPA No. D10).

Pediatric rheumatology rotations are an essential component of the Saudi Rheumatology Fellowship Curriculum⁶ as well as among other rheumatology programs.¹⁴ We composed an initial EPA addressing pediatric presentations which stated “recognizing and triaging presentations of common pediatric rheumatologic diseases.” There were two pediatric rheumatologists among our experts. Despite that, this particular EPA did not stand following the first round of expert voting. This might represent a great discrepancy in the current practice of rheumatology in Saudi Arabia. According to Ministry of Health regulations,²⁶ any patient older than 14 years should be assessed and managed by an adult specialist, including those with rheumatological diseases. The pediatric age in most countries ends at 18 years.²⁷ This means that all adult rheumatologists in Saudi Arabia are expected to assess, diagnose and manage pediatric patients age 14 to 18 presenting with common rheumatological diseases. Our EPA addressing pediatric rheumatology was related to recognizing and triaging presentations of common pediatric rheumatologic diseases, but it did not specify anything related to initial diagnosis and management.

Our experts also emphasized the value of procedures, particularly EPA No. C3, which is related to knee joint aspiration and injection. Clearly, knee joint aspiration and injection can be embedded under EPA C1 and EPA C2, but our experts agreed to make it a separate EPA. This is a strong message to all stakeholders addressing tasks and responsibilities of graduating rheumatologists. In addition, this leads to the discussion of crossing boundaries across different specialties.¹⁷ EPA No. C3 can also be addressed in other specialties like family medicine and internal medicine. The same concept can be applied when addressing EPA No. A8. Radiologists have obviously undergone more dedicated training than rheumatologists in interpreting musculoskeletal (MSK) imaging.

Musculoskeletal ultrasonography (MSKUS) is not addressed in our EPAs. The Saudi Rheumatology Fellowship Curriculum⁶ did not state any competency related to performing MSKUS. It is known that performing MSKUS is part of the competencies of graduating rheumatologists in some European programs.^28,29 We think that MSK examination skills should have more emphasis in our training, rather than relying on imaging modalities.³⁰ Physical examination will remain the most common diagnostic tool used by doctors.³¹ MSK examination skill deficiencies are well reported in our trainees³² and elsewhere.³³ Imaging might have a benefit in early diagnosis^34–36 but offers no superiority over standard MSK assessment in monitoring diseases^37–39 and might actually cost more without added benefit.⁴⁰

There are several limitations. This research work is specific to Saudi Arabia and may not apply to other countries. The findings here are great tools for CBME design and implementation, yet they are not definitive. This research should be taken as a continuum of a rubric where mapping of these EPAs should take place. The field of clinical practice is changing rapidly, with new approaches in diagnosing and treating rheumatological diseases. In addition, a new era may arise due to the extreme pressure from the surrounding health environment, like what happened as a repercussion of all issues related to autoimmune phenomena associated with COVID-19 infection and its vaccines. Considering all of that, we highly recommend frequent assessment of these EPAs by committees supervising any training program and considering appropriate adjustments and/or modifications to match the current trends in health care practice for rheumatology.

Our work here adopted a strict methodology to develop and validate a core set of EPAs for rheumatology fellowship training programs in Saudi Arabia. We hope this represents an added value to shape the current training and assessment in rheumatology fellowship programs. Further specifications of each EPA with adequate mapping and milestone development are essential for proper development of workplace curriculum.