Evaluation of Applying Standardized Patients in Certification Assessment of Clinical Practice Ability Standardized Training for Psychiatric Resident Physicians

Since the concept of standardized patients (SP) was proposed by Professor Barrows and Abrahamson in 1964, it has been extensively applied in skills training, assessment, and development in medical education in many countries.¹ SP refers to, after systematic training, the ability to accurately and realistically portray the clinical symptoms and signs of patients, serving as a triple identity of patients, evaluators, and respondents. SP can be healthy people or patients.² Due to the particularity, operability, and safety issues of patients, the assessment of clinical practice ability in psychiatry usually utilizes the advantages of SP to replace patients. The combination of SP and objective structured clinical examination (OSCE) has been widely used in the training and assessment of resident physicians and the clinical skills examination in practicing physician assessment in the United States, Canada, and the United Kingdom. This includes psychiatric examination stations.³ SP overcomes the difficulty of finding highly characteristic cases and situations in the previous clinical practice of psychiatry while creating a safe atmosphere, which is a substantial breakthrough in psychiatry education.

Currently, China adopts a combination strategy of “regular assessment”, “phase assessment”, and “graduation assessment”. The regular assessment uses comprehensive assessment methods, including the mini-clinical evaluation exercise (Mini-CEX)⁴ and direct observation of procedural skills (DOPS),⁵ listed as the regular assessment score registration system. The phase assessment utilizes the form of the national practicing physician practical skills examination to regularly conduct skill practice operations and comprehensively evaluate students’ clinical skills and thinking abilities in psychiatry. The graduation assessment employs the OSCE mode, investigates and collects medical histories with multiple stations, and physically checks basic skills. The graduation assessment is an indispensable part of quantifying clinical competence and evaluating job competency. Moreover, the use of SP plays a crucial role in assessing evaluation consistency and stability.⁶ In 2020, the Health Human Resources Development Center organized experts to revise the guidance standards for the assessment of practical skills in residential training graduation and launched a new “Clinical Practice Ability Assessment Standard Plan for Standardized Training Graduation of Resident Physicians (Psychiatry)”. However, the evaluation of the application of the assessment plan is insufficient, there has been no research or evaluation on the implementation effect of this plan. This standard sets up one general examination station for cardiopulmonary resuscitation and four independent examination stations, including clinical skills, clinical thinking, clinical communication, and neurological examination. The Affiliated Brain Hospital of Guangzhou Medical University is the first specialized hospital for mental illness, the earliest clinical and teaching base for psychiatry, and one of the five national standardized training bases for psychiatric resident physicians in China. It has undertaken the graduation assessment of practical skills for psychiatric resident physicians in Guangdong Province in recent years. Since the revision of the assessment standard in 2020, it has conducted assessments combining SP with OSCE and accumulated some experience. This article mainly analyzes and summarizes the feedback on the application of SP in the certification assessment of practical skills in residential training, aiming to provide a valuable reference for peers and contribute to the continuous improvement of the quality of certification assessment of standardized training for psychiatric resident physicians.

Subject and Method

Subject

Candidates and examiners who participated in the certification assessment of clinical practice ability standardized training for psychiatric resident physicians in Guangdong Province from 2020 to 2023 were incorporated in this study as the research participants.

Method

According to the 2020 assessment standard, a multi-station assessment form was adopted, which is more relevant to psychiatry. A specific exam execution team was established to coordinate the design, preparation, and implementation of the assessment based on the characteristics and actual situation of the profession. Four examination stations were set up, including clinical skills, clinical thinking, clinical communication, and neurological examination. Among them, the first, third, and fourth stations had medical students/doctors playing the role of patients/patient families. Each station was evaluated synchronously and independently by two examiners from different professional bases who had received unified examiner training and obtained the Guangdong Province Graduation Practice Skills Examiner Certificate. The average score was taken as the assessment result. Measured by the percentage system, each project was calculated based on weights and included in the total score, with a maximum score of 100 points. The assessment content is shown in Table 1.

Table 1 Information on Clinical Practice Skills Certification Assessment Stations

The medical students/doctors impersonating patients/family members at each skill assessment station had received unified training. The clinical skills station included psychiatric examination, medical history collection, and necessary physical examinations. During psychiatric examination, medical students and doctors impersonated patients and family members, respectively. The clinical thinking station contained case analysis, question answering, and medical record summarizing, mainly involved in symptom and syndrome analysis, diagnostic analysis, diagnostic conclusions and criteria, risk assessment and management measures, and treatment. In the clinical communication station, examiners acted as the family members of patients and communicated with candidates based on the characteristics of relevant cases. Regarding the neurological station, candidates conducted specialized physical examinations of the nervous system on patients played by medical students.

Questionnaire Survey

After the assessment and before the announcement of the results, an online anonymous questionnaire survey was conducted on the examiners and candidates, including the rationality recognition of the exam execution, content, difficulty, and form of each station, overall satisfaction, and, most importantly, the evaluation of SP. The assessment satisfaction questionnaire set ten items for the examiners and candidates, respectively. The SP satisfaction questionnaire was composed of 20 items, of which the first ten were positive questions, followed by ten negative questions. All options were scored according to the Likert level 5, with 5 points representing extremely satisfactory/completely agree and 1 point denoting extremely unsatisfactory/completely disagree.

Statistical Processing

The data were statistically analyzed using SPSS 26.0 and GraphPad Prism 8.0. Adobe Illustrator was adopted for image processing. The scores of stations were described with mean ± standard deviation. The Kruskal–Wallis rank-sum test was employed for statistical analysis, with P <0.05 as the statistically significant difference. The questionnaire answers were described with percentages. The Kendall test⁷ was used to analyze the consistency of the questionnaire responses of the two groups, with P <0.05 as the statistically significant consistency. The Kendall coordination coefficient <0.2 represents poor consistency, 0.2–0.4 implies moderate consistency, 0.4–0.6 indicates moderate consistency, 0.6–0.8 represents strong consistency, and 0.8–1.0 indicates strong consistency. The Wilcoxon rank-sum test was employed to explore the differences in the questionnaires between the two groups, with P <0.05 indicating a statistically significant difference.

Results

Overall Situation of Practical Skills Certification Assessment

From 2020 to 2023, there were 523 candidates in total, including 187 males and 336 females, with an average total score of 83.88 ± 5.78. The average scores for each station are 83.77 ± 6.80, 81.50 ± 6.81, 84.46 ± 6.70, and 88.31 ± 7.82, respectively. The assessment scores for the remaining years are shown in Table 2. Except for the second station, there are significant differences in assessment scores among the other stations in different years.

Table 2 Summary of Practical Skills Certification Assessment (Mean ± Standard Deviation, Ranging from 0 to 100)

Evaluation of Satisfaction Degree of Examiners and Candidates with the Certification Assessment of Practical Skills

Valid questionnaires collected from examiners and candidates are 47 and 138, respectively, as shown in Table 3 and Table 4. Those who answered “Extremely Satisfactory” or “Satisfactory” are considered satisfied with the content of the items. According to the results, 89.36% of the examiners and 85.51% of the candidates have overall satisfaction with the examination execution. The satisfaction degrees of the examiners with the examination environment and equipment are lower than the overall satisfaction (87.23%, 85.11%, and 89.37%). The candidates have lower satisfaction with the examination environment, form, difficulty, and scope than the overall satisfaction (78.99%, 83.33%, 78.99%, 84.06%, and 85.51%).

Table 3 Evaluation of the Examiners’ Satisfaction with Practical Skills Certification Assessment (%)

Table 4 Evaluation of the Candidates’ Satisfaction with Practical Skills Certification Assessment (%)

Consistency of SP Satisfaction Evaluation of Examiners and Candidates

The Kendall test was adopted to analyze the consistency of SP satisfaction evaluation of examiners and candidates. As shown in Table 5, the consistency of evaluation is statistically significant, with the examiners’ consistency being higher than that of the candidates. The Kendall coordination coefficient of the examiners is 0.829, indicating high consistency, while that of the candidates is 0.012, representing low consistency.

Table 5 Consistency Test of SP Satisfaction Evaluation

Specific Evaluations of SP by Examiners and Candidates

The questionnaire of SP evaluation for examiners and candidates contained 20 items, of which the first were ten positive questions, followed by ten negative questions, as shown in Table 6. The intergroup comparison exhibits seven items that the satisfaction degrees of examiners and candidates are inconsistent (examiners vs candidates, the ratio of strongly agree + the ratio of agree), as shown below: tally with the appearance of patients (100% vs 92.75%), the patients proficiently respond to medical history information with appropriate pronunciation and intonation (98.08% vs 94.21%), conform to patient physical movements (98.08% vs 93.48%), patient performance is consistent and coordinated (100% vs 95.65%), the patients accurately express the language (96.16% vs 94.21%), the patients accurately express psychological status through language (98.07% vs 96.37%), and the patients use medical terminologies (17.31% vs 0%). Among the seven items, the first six are positive, while the last one is negative. Although the satisfaction ratios suggest that the examiners and candidates have consistent trends, the former is more satisfied with the appearance of patients, voice and intonation, physical movements, coherence, and language accuracy. However, in terms of whether to use medical terminologies, compared with the candidates, the examiners tend to believe that there is a use of medical terminologies in SP.

Table 6 Evaluation of SP Satisfaction of Examiners and Candidates (%)

Discussion

Huang et al⁸ disclosed that the overall prevalence of adult mental disorders in China was 17%, with approximately 16 million patients suffering from severe mental diseases, and the number of registrations had been increasing year by year. Mental health has attracted social attention. Meanwhile, the lack of professional talents has constrained the development of the mental health industry. Therefore, it is of significant importance to cultivate practical talents in mental illness and mental health that meet social needs, centering around the construction of clinical thinking and practical abilities. Clinical practice is also the continuation and perfection of theoretical knowledge.⁹ In 2020, the Health Human Resources Development Center of China organized experts to revise the guidance standards for the graduation assessment of practical skills in residential training and launched a “Clinical Practice Ability Assessment Standard Plan for Standardized Training Graduation of Resident Physicians (Psychiatry)”. However, there has been no application evaluation of the assessment plan yet.

As the major undertaking institution for the practical skills assessment of psychiatric resident physicians in Guangdong Province, the Affiliated Brain Hospital of Guangzhou Medical University has accumulated some experience and participated in the formulation of the assessment plan. By analyzing the results of the certification assessment in the past four years and the satisfaction degree with the assessment, from the perspective of the application of SP, it is found that (1) the scores of stations involving substitute patients and substitute family members fluctuate greatly. The standardization level of SP is essential to the stability of the certification assessment; (2) Examiners and candidates are generally satisfied with the graduation exam. The exam environment and equipment significantly impact examiners, while there is still room for improvement in the satisfaction of candidates with the exam environment, format, difficulty, and scope; (3) Examiners have consistent evaluations of SP, while the feedback from candidates is not harmonious; (4) The appearance, intonation, physical movements, coherence, and language accuracy of SP are crucial factors that influence the experience of candidates, and the use of medical terminologies by SP is a key observation point for examiners. We speculate that the above findings may be related to factors such as the candidates’ mentality before the examination (for example, eagerly wanting to obtain information supporting diagnosis from the medical history description of SP), the candidates’ shorter contact time with SP than the examiners’, and the examiners’ potential role as family members. The significance of this study is not only to understand the application of SP under the new assessment standard scheme but also to reveal the difference in experience between examiners and candidates in the current graduation assessment mode. The examination environment, hardware facilities, unifying examination difficulty, and examination scope can be rectified in the short term. However, the investment in SP rectification is enormous, making it difficult to shorten the gap between SP and actual patients. Personnel with medical backgrounds are prone to using medical terminologies, and not every doctor can be a good actor. SP training requires investing funds, energy, and time, which is costly. Unpaid SP has low persistence and is adverse to the stability and protracted nature of standardized teams.

Meanwhile, future directions should focus on the following issues: (1) Psychiatry emphasizes empathy,¹⁰ and whether the implementation of SP will weaken students’ empathy ability; (2) Since SP mostly presents classic cases or single diseases, will it affect students’ identification of comorbidities and rare diseases; (3) Whether the process of cultivating and assessing standardization restricts the divergent thinking of students; (4) This study evaluated the application situation from the perspectives of examiners and candidates and found differences between the two. Should subsequent research take the experience of SP in assessment as one of the observation subjects?

In summary, the OSCE assessment model using SP receives a high degree of satisfaction in the certification assessment of the standardized training for psychiatric resident physicians. The exploration of teaching or assessment models should adhere to the patient-centered strategy, construct rigorous clinical thinking in the cycle of discovering and solving problems, and advance independent analysis ability and clinical diagnosis and treatment level. Continuously identifying, proposing, and addressing problems in practice, supplemented by the cultivation of scientific thinking, can inject new vitality into the development of medicine. A complete and systematic assessment system is necessary and meaningful in ensuring the quality of training.