A Comparison of Dental Students’ Self-Assessment and Instructors’ Assessment in Competency Examinations in a Preclinical Operative Dentistry Course

Introduction

Higher education institutions are increasingly promoting lifelong learning among students.¹ The students’ ability to demonstrate evaluative judgment, that is the capability to make decisions about the quality of work of self and others, is an essential element in fostering self-directed learning and academic growth.² It is a skill developed through a range of activities in a process that contribute to the augmentation of students’ evaluative abilities. Student self-assessment is a known strategy that supports the development of evaluative judgment skill.² Self-assessment practices are taught and learned, involving engaging the students in evaluating their own performance compared to established standards and criteria while attempting assigned assessment tasks.^3,4 In health science education, self-assessment enables students to develop the ability to reflect on challenges and interactions in real-world practice.⁵ Therefore, it is expected that student self-assessment practices are integrated into health sciences education curricula to meet accreditation bodies’ requirements and equip students with the attribute of seeking lifelong learning upon graduation.⁶ As with any learning tool, self-assessment should be valid and accurate to facilitate continuous performance development. Furthermore, accurate self-assessment is argued to be a requirement for effective self-reflection, an essential attribute that promote and sustain students’ learning.⁷ Given the important role self-assessment ability plays in empowering lifelong learning, it is of significant importance to evaluate the capacity and performance of student self-assessment in terms of accuracy to ensure they are guiding the students effectively to address deficiencies and maintain competency levels.

In dental education, students’ performance is judged based on their ability to acquire a set of taught skills in the cognitive (knowledge), affective (attitude), and psychomotor (manual skills) domains.⁸ The development and acquisition of dental students’ skills in the psychomotor domain have received great attention in the curricular structure to ensure successful clinical practice. In dental curricula, psychomotor learning activities are introduced early in undergraduate dental programs, typically preceding clinical learning in practical and simulated clinical settings. In simulated clinics, teaching instruction is often structured with a focus on simulated clinical procedures and methods for task performance, thereby facilitating the development of psychomotor skills. A teaching team methodically assesses students’ performances in preclinical tasks, ideally using an evaluation rubric, which is a known tool designed to support evaluation of performance with high consistency and promote learning by introducing the assessment expectations of given task.⁹ Considering the significance of students’ evaluative judgment skills, early integration and practice of self-assessment strategies in dental education is crucial for improving students’ ability to recognize ideal and non-ideal outcomes, thereby positively influencing their lifelong learning journey.⁵ The implementation of self-assessment in preclinical dental curricula has been studied across various dental disciplines. In the context of operative dentistry courses, researchers have explored the role of student self-assessment practices in developing their evaluative judgment skill and performance in relation to students’ factors, including gender, that could contribute to their success in preclinical operative dentistry. A recent study by Sadky et al found that male students tend to overestimate their abilities, while it is often underestimated by their female counterparts.¹⁰ The gender-related difference in respect to student self-assessment in this learning domain is also studied by Kornmehl et al, suggesting that this factor may affect the accuracy and performance of self-assessment by students.¹¹ Other studies by Mittal et al and Ciardo et al highlight the significant positive impact of self-assessment on enhancing students’ learning in preclinical dentistry courses, when feedback and clear assessment criteria are applied.^12,13 The performance of students in self-assessment has also been found to be related to the training level, as reported by Huth et al, who noted the improved agreement between student and instructor assessments with increased training.¹⁴ Similarly, the experience has been linked to the accuracy of self-assessment, as demonstrated by Metz et al and Tuncer et al, who observed the increased alignment between student and instructor assessments in subsequent years.^15,16 Nevertheless, in spite of these findings, it remains unclear how differences among dental students can impact the accuracy and improvement of their self-assessment abilities. As such, studies evaluating the impact of gender-related differences on self-assessment accuracy have yielded varied results.^17–20

The inherent differences among students are acknowledged in education, including gender and achievement levels, and can impact their self-assessment accuracy, and thus their evaluative judgment abilities. High-achievers may be more accurate self-assessors owing to their better understanding of assessment standards, while low-achievers may show a tendency toward deviated estimation of their actual performance evaluation.²¹ The impact of gender differences on self-assessment ability among students are often driven by cultural influences, that can shape the way of evaluating their abilities. Males in some cultures are expected to demonstrate confidence, which may result in overestimation of performance in self-assessment. In the same context, females in contrast to males, tend to perform more conservative evaluations during self-assessment.¹⁰ Moreover, stress encountered more by females may negatively impact their confidence level, and consequently their ability to self-assess their work objectively.²²

At King Saud bin Abdulaziz University for Health Sciences (KSAU-HS), the preclinical operative dentistry course is offered in the first year as part of the undergraduate dental program, available to both male and female students. This course is designed to impart the necessary knowledge and psychomotor skills for performing clinical procedures in operative dentistry. Self-evaluation is introduced in the first year of study and practiced throughout the course utilizing structured rubrics. Considering the significant influence of gender differences and the achievement levels on self-assessment performance, it would be beneficial to explore the degree of self-assessment ability in relation to achievement levels of male and female students compared to their actual performance derived from instructors’ assessment during competency examinations in a Preclinical Operative Dentistry course within this geographical area. Evaluating how these factors interact during self-assessment will help underscoring the need to use considerate approaches in learning practices, that account for diverse student experiences and support fair opportunities for effective evaluative judgment development.

Therefore, this study aims to compare dental students’ self-assessments and instructors’ assessments in competency examinations and evaluate self-assessment improvement in a Preclinical Operative dentistry course at KSAU-HS University. The first null hypothesis was that there would be no difference between students’ self-assessment scores and instructors’ evaluation scores. The second null hypothesis was that there would be no difference in students’ self-assessment ability across two successive competency exams: the intra-semester and the final preclinical operative dentistry exams. The third was that the students’ genders and achievement levels would not predict their self-assessment ability.

Materials and Methods

Approval for conducting the study was obtained from the Institutional Review Board at King Abdullah International Medical Research Center (IRB# NRC22R/616/10). The study was conducted in the College of Dentistry, KSAU-HS, using the assessment records from the preclinical Operative Dentistry I course of the academic year 2020–2021. The Operative Dentistry I course was offered during the first year of study in the undergraduate Doctor of Dental Medicine program for batches of male and female students. The course was coordinated by one of the co-authors (MAA) and took a role in course instruction, assessment process, and collecting and storing the assessment records of practical examinations. To minimize bias, the retrospective data were extracted from the stored assessment records and de-identified for the study using a data collection sheet independently by another co-author (SM), who was not involved in course instruction and assessment. Moreover, the data was statistically analyzed by another member of the research team (AA) who was not involved in the course. In the academic year from which the study’s data is retrieved, the course enrolls 44 students in male batch and 43 students in female students’ batch, with eight instructors contributing to each batch.

For all preclinical simulation exercises, students received formal written instructions via the learning management system’s announcement messages and during the first lecture on the evaluation and grading criteria for each step of the planned procedures. The rubric for each planned procedure was made available to all students and instructors through the coursebook, evaluation sheets, and announcements. The course coordinator calibrated the grading method among the instructors before starting the course. In the calibration session, the instructors are provided with information on the grading rubrics, teaching team’s expectations of student performance with discussing and agreeing on specific guidelines for teaching and assessment activities. Following the discussion in the calibration session, instructors participated in sample grading exercise to ensure that the application of the assessment tool and the evaluation outcomes are in harmony among all instructors. This calibration activity for instructors is a usual practice in the course conducted each academic year.

The instructors also taught and graded students’ performances in all continuous simulation exercises in the course. In the competency examinations, each step of the restorative procedure performed by each student was assessed independently by two calibrated instructors, one of whom was a full-time instructor. The average value of the two instructors’ scores for each student was calculated and recorded as that particular student’s grade. Self-assessment of the practical exercises was introduced to students early in the course and was voluntary, with students encouraged to engage. Before each competency examination, students were instructed to evaluate each assessment item regarding their work and provide a grade based on selected rubrics and criteria before presenting it for evaluation by the instructor. Students completed both an intra-semester and a final examination as competency tests in this course, where they provided self-assessments of their performance using the same rubric that was used by the instructors. Table 1 presents the required restorations and their associated assessment items in the intra-semester and final examinations.

Table 1 Assigned Procedures and Criteria for Each Assessment Item With Grade Distribution Listed in the Rubrics of the Intra-Semester and Final Examinations

Data collected from the assessment records comprised both instructors’ scores and students’ self-assessment scores. Instructors’ assessment records that lacked any corresponding student self-assessment records were excluded from the study.

Data were analyzed using SPSS. Inter-rater reliability and intra-class correlations (ICC) were used to assess the agreement level among the scores provided by instructors. The Mann–Whitney nonparametric test was employed with a significance level of 0.05. Comparisons were made between students’ self-assessments, instructors’ assessments, and their level of agreement in the intra-semester and final examinations. The agreement level of the assessments was evaluated using inter-rater reliability estimates.

Results

The assessment data of 89 students (50.6% of whom were male students) were collected and analyzed. The data included assessment scores (total and sub-scores) recorded by students and calibrated instructors concerning the preclinical operative dentistry procedures performed by students during the intra-semester and final examinations.

Instructors’ Assessment

A post hoc ANOVA test applied to the total assessment scores revealed no significant difference (p = 0.949) in the mean scores recorded by full-time and part-time instructors, indicating that the instructors had successfully achieved calibration in the course.

The overall agreement of the instructors’ scores found to be significantly excellent (p < 0.001) between full-time and part-time instructors in the intra-semester and final examinations (ICCs = 0.888 and 0.859, respectively) (see Table 2).

Table 2 Reliability Evaluation With Intra-Class Correlation Coefficients (ICCs) for Scores of Full-Time and Part-Time Instructors in Intra-Semester and Final Examinations

Students’ Self-Assessment Performance

Gold Standard

The average score of the two instructors was used as a gold standard to evaluate the agreement level between students’ self-assessment scores and instructors’ grading scores. A post hoc ANOVA test conducted on the total assessment scores revealed a significant difference (p < 0.001) between the mean scores of instructors and students.

Agreement

A significantly moderate agreement (p < 0.001) was observed between students’ self-assessment scores and full-time instructors’ (ICC = 0.488) and part-time instructors’ scores (ICC = 0.436). Table 3 depicts a decline in the agreement between students’ self-assessment scores and full-time faculty’s scores from the intra-semester examination to the final examination.

Table 3 Reliability Assessment With Intra-Class Correlation Coefficients (ICCs) for Student Self-Assessment and Instructor Assessment Scores in Intra-Semester and Final Examinations

The students rated their work higher than the faculty members did. The mean difference between the student self-assessment scores was significantly higher than the scores given by full-time instructors by 0.43 points and those given by part-time instructors by 0.46 points (Figure 1).

Figure 1 Estimated marginal means of the scores of students’ self-assessment, full-time instructors’ assessment, and part-time instructors’ assessment in two examinations: intra-semester and final examinations.

Students’ Gender and Achievement Level as Predictors of Self-Assessment Performance

Figure 2 illustrates the results of students’ self-assessment performance based on gender. Chi-square tests revealed a significant gender association in the intra-semester examination (p = 0.048). Repeated-measures ANCOVA revealed that the difference in the rate of the total scores from the first examination to the second examination had a significant interaction effect with gender. Male students’ self-assessment evaluations were approximately 60% higher than the instructors’ average score. However, female students evaluated themselves more reasonably, with 44.3%, 39.8%, and 15.9% having accurate, high, and low estimation levels of evaluations, respectively. The chi-square tests also revealed a significant association between self-assessment performance and the achievement level (p = 0.030). Repeated-measures ANCOVA revealed that the initial student self-assessment, as a covariate, explained the significant amount of variance in the dependent measure—the final grade. Figure 3 shows that approximately 60% of students who overestimated their performance during the intra-semester examination tended to improve their actual performance, receiving higher instructor scores in the final examination. By contrast, students whose self-assessed scores were lower than those assigned by instructors in the first examination did not demonstrate improved scores in the second examination.

Figure 2 Students’ self-assessment performance in the intra-semester examination by gender.

Figure 3 Impact of students’ self-assessment performance on their achievement improvement in the final examination.

Table 4 presents comparisons between male and female students in terms of their mean (SD) self-assessment scores and the agreement levels with the instructors’ scores in the intra-semester and final examinations. When each assessment item was analyzed independently, both male and female students’ self-assessments showed significantly good agreement (p < 0.05) with instructors’ assessment scores for a rubber dam application procedure and cavity preparation, as well as total scores for the intra-semester examination. However, the agreement between the instructors and the male and female students was not significant for scoring amalgam restoration and operator position during the intra-semester examination. In the final examination, the level of agreement between instructors’ scores and students’ self-assessment scores differed between male and female students. For assessment scores of the Tofflemire matrix and rubber dam application procedures, female students had a significantly good level of agreement (p < 0.05) with instructors. Male students demonstrated a fair level of agreement with the instructors for these assessment items. Male students demonstrated a good level of agreement with instructors for scoring cavity preparation in the final examination, as well as for the total scores. Female students demonstrated a relatively low level of agreement with instructors for these assessment items.

Table 4 The Degree of Agreement Between Students’ Self-Assessment Scores and Instructors’ Assessment Scores by Students’ Gender (Male Versus Female)

The mean (SD) of students’ self-assessment scores and instructors’ assessment scores, as classified based on students’ achievement levels in examinations, are presented in Figure 4. As students’ achievement levels decreased to “good” and “fair”, there was a corresponding downward trend in their self-assessment scores, with a significant increase in the difference between their scores and instructors’ scores. However, among high achievers, there was no significant difference in scoring ability between instructors and students classified under the “excellent” achievement category (p = 0.392).

Figure 4 The degree of agreement between students’ self-assessment scores and instructors’ evaluation scores based on students’ achievement levels.

Discussion

Self-assessment is a critical skill that dentists must acquire in order to be competent oral healthcare providers.²³ A dental student must be able to self-evaluate their strengths and weaknesses accurately to become a lifelong, self-regulated, and successful dental practitioner.¹⁶ Recognizing its importance, many health education regulatory bodies mandate competency in self-assessment.²⁴ Students’ preclinical performance is correlated with their self-assessment accuracy, and these go hand-in-hand. As students practice and hone their psychomotor skills, their self-assessment skills are expected to improve proportionally.²¹ Therefore, this study was undertaken in a preclinical operative dentistry setup, where psychomotor and self-assessment skills were introduced in the undergraduate dental program.

Calibration is crucial for achieving successful student self-assessment and self-directed learning.¹⁵ A few studies have reported that faculty members grading students are calibrated.^16,25–28 In our study, the course coordinator calibrated the grading method of the full- and part-time instructors before the course began. No significant difference was found in the mean scores recorded by full-time and part-time instructors. Excellent agreement in assessment scores between instructors for both intra-semester and final examinations indicated successful calibration of instructors’ grading method. Consequently, instructors’ assessments served as the standard for evaluating students’ self-assessment performances.

An assessment using a well-structured rubric supports learning and facilitates feedback and self-evaluation.²⁹ A grading rubric provides clear expectations and assessments to both students and faculty.^9,30 Nevertheless, a systematic review of the use of self-assessment in preclinical/clinical dental education highlighted that only a few studies mentioned employing some form of criteria sheet to grade students’ performances.⁵ We employed a grading rubric for assessing all lab exercises and competency examinations, finding it had great potential to promote students’ learning and self-assessment.

Our results revealed a significant difference between instructors’ and students’ mean evaluation scores, leading to the rejection of the first null hypothesis. Similar findings have been reported in previous studies.^10,15,31 This finding may stem from students’ lack of competency in assessing the items during the early stages of the academic program. Previous studies have found that the more difficult the task, the greater the discrepancy between self-assessment and faculty assessment.^32,33 This finding aligns with the results of our study, as showed a decline in the agreement level between students’ self-assessment scores and instructors’ scores from the intra-semester examination to the more challenging final examination.

Self-assessment had a predictable impact on the progress of students’ performance. Improvement in self-assessment was the result of thorough practice and experience. Furthermore, in a study involving a complete denture course conducted over two consecutive laboratory examinations, Curtis et al suggested that improving students’ ability to make accurate self-assessments could lead to performance progress.³⁴ Furthermore, researchers have observed modest improvements in student–faculty agreement when the study design includes multiple self-assessment trials.^34,35 We obtained similar results when the self-assessment was evaluated in two consecutive competency examinations—the intra-semester examination and the final examination. Approximately 60% of those with overestimated self-assessment evaluations in the intra-semester examination improved their actual performance in the final examination. By contrast, students who assigned themselves lower scores compared to the instructors’ evaluation in the first examination did not obtain improved scores in the second examination. Thus, our second null hypothesis was rejected. On other hand, Tuncer et al compared the assessment scores of second- and third-year dental students and the faculty in two consecutive preclinical practical examinations during restorative dentistry courses and reported no improvement in students’ self-assessment ability even though there was a noted improvement in their skills between the successive exams.¹⁶

Gender served as a lens to explore students’ self-assessment performance. Male students tend to overestimate their performances in self-assessment than female students during the intra-semester examination, and overall, female students evaluated themselves more accurately. Thus, our third null hypothesis was rejected, aligning with previous studies indicating that female students do not overestimate themselves compared to their male counterparts in self-evaluation.^10,11,36 However, Hadid did not observe gender effect in students’ self-assessment performance.¹⁸ In our study, gender differences were evident in how students self-assessed each step of the procedure. In the intra-semester examination, both male and female students had a significantly good level of agreement with the instructors regarding assessment scores for the rubber dam application procedure and cavity preparation, as well as in the total scores. In the final examination, male and female students differed in their level of self-assessment-related agreement with the instructors’ scores. The discrepancy between male and female self-assessment performances during the second examination could be attributed to the increased difficulty of the tasks in the final examination; this effect has been reported previously.^32,33

Our current study demonstrated that student achievement levels can predict students’ ability to accurately self-assess their performance. We found that the self-assessment scores of students who attained the highest level of achievement in the examinations tended to correspond closely with those of their instructors. As the actual performance level decreases, there is an expected decrease in the level of agreement between students’ self-assessment and instructors’ assessment. Compared to the existing literature on preclinical operative dentistry, this study is one of the few that specifically analyzes each competency assessment item in an evaluative context and uniquely evaluates the impact of gender on each self-assessed item. It is important to note that this study focused on preclinical dental training courses conducted during the early stages of the academic program. Thus, generalizing the current study’s findings will necessitate data from advanced operative dentistry courses in both preclinical and clinical stages; this is a research avenue that is worth exploring in the future.

Conclusions

Within the context of preclinical dental education of operative dentistry, the outcome of students’ self-assessment of their work at this early level of dental education is generally not in alignment with their instructors’ assessments outcome. When the outcome is viewed progressively, students with overrated self-assessment evaluations tend to improve their actual performances, while those who underestimated their performance during self-assessment tend to not demonstrate improvement in the actual performance. Furthermore, students’ level of achievement and gender moderate their self-assessment abilities. Future studies must explore the development of students’ self-assessment abilities alongside the transformation of their knowledge, skills, and values throughout the academic program.