Evaluating the Transformative Impact of Online Education on Medical Student Learning Outcomes

Introduction

COVID-19 was first discovered in December 2019 in Wuhan, China. Soon after, the World Health Organization announced Covid-19 as a global pandemic. Every country implemented public health strategies to control the transmission of the virus, including handwashing, wearing face masks, hand cleaning, physical distancing, and preventing gatherings. There were more strategies applied in March 2020 to flatten the curve and control the transmission of COVID-19, such as lockdowns, staying-at-home strategies, school and university closures, and a reduction of business hours.¹ In very country, Covid-19 has affected all sectors at some level, including health, economics, education, and tourism. However, education is considered the most affected sector by the Covid-19 pandemic. During the lockdown phases, schools and higher education entities used varying levels of online learning. These levels range from total online learning to just minimizing class attendance.²

Prior to the Covid-19 pandemic, online learning was known and used in very limited circumstances. However, the Covid-19 pandemic has paved the way for online learning to become widely used.³ In any difficult situation, online learning can assist schools and universities to continue and facilitate student learning.⁴ The dynamics of online learning in higher education have shifted, moving away from a model where communication is primarily one-way, driven by the instructor, towards a two-way communication system, placing a greater emphasis on learners actively engaging with technology tools.⁵

Online learning has also been recognized as providing more accessibility to learning for physically challenged students with limited movement and those undergoing psychological or emotional distress.^2,6 Globally, many institutions offer online learning to solve other challenges, such as learning costs, staff shortages, and student circumstances. However, the effects and efficacy of online learning are still not well understood.⁷

Education using online learning involves various platforms and can be adapted according to the educational needs synchronously or asynchronously. Some of the online platforms provide communication and collaboration between educators and learners, for example, Microsoft Teams, Google Classroom and Blackboard. They allow teachers to create educational and training programs through the use of many features such as workplace chat, video meetings, and the sharing of different content like Word and Excel files. Teachers can also assess their students in different ways, such as through quizzes and rubric-based assessments.² Importantly, online learning effectively encourages and improves some skills, including problem-solving, critical thinking and self-directed learning. Finally, online learning provides learning resources by recording classes and saving them.^2,8 On the other hand, Internet connection and access to digital devices are major issues of online learning in many countries around the world. For example, in Pakistan, a significant deal of learning and teaching, as well as administrative activities of academic institutions, are handled manually.⁹ Lack of access to fast, affordable and reliable internet connections hinders the process of online learning, especially for those who are living in rural as well as marginalized communities of Pakistan.¹⁰ Moreover, interaction between educators and learners might not be efficient enough, especially where course content is sent by e-mail, which requires response time.¹¹ It has also been noted that student assessment during online learning is subject to error, uncertainty, and confusion among educators and learners.² Finally, the extent of parental guidance and concerns over increased screen exposure are issues for educating young learners online.

There is a need to evaluate and assess online learning strategies at all levels. A recent bibliometric analysis has revealed that the majority of research in online learning has been published since 2020, that is, after the start of the Covid-19 pandemic.¹² Researchers have begun to examine certain aspects of online learning, including e-learning teaching infrastructure, limitations of online learning, and non-conducive learning environments.²

Many studies have focused on the main challenges and factors influencing e-learning success. These challenges have been classified into four categories: technological challenges (eg lack of technological infrastructure), individual challenges (eg lack of IT skills of users), cultural challenges (eg political and economic situation of students) and course challenges (practical subject). These challenges are different from one country to another country in both developed and developing countries.¹³ For example, the Al-Araibi et al study has found that technological issues are the main criteria for the successful online learning system in developing countries where only 15% of online learning projects are successful. In comparison, 45% of online learning projects are total failures, and 40% are partial failures due to technical issues.¹⁴ Another study has found that online learning structure was affected by student autonomy, student’s background, students’ prior experience, student-instructor dialogue, and student-student dialogue.¹⁵ Furthermore, an interview study showed that self-learning is essential of online learning improvement.¹⁶

The technology acceptance model (TAM) is an information systems model to measure peoples’ tendencies or readiness to use innovative technologies in both learning and lifestyles. TAM has been used to analyze the effectiveness and easiness of use of online learning models. This model has shown dramatically increasing interest and growing attention in online learning research. It has found that the top 10 countries in the online learning field of research are Malaysia, The United States, China, Taiwan, Indonesia, United Kingdom, Saudi Arabia, Australia, India, and Canada. Moreover, current research demonstrates the tendencies and trends in accepting online learning, as well as the models employed in prior studies, which can be categorized into eight significant models: the technology acceptance model (TAM) (1/2/3), The Task Technology Fit (TTF), The Information System Success Models (ISSM), The general extended technology acceptance model for e-learning (GETAMEL), the Expectancy Confirmation Model (ECM), The Technology Acceptance Model (EAM), Unified Theory of Acceptance and Use of Technology (UTAUT).

Current research also demonstrates that the main patterns in online learning settings include blended learning (discussed below), E-learning (LMS), flipped classes, web-based videoconferencing, and Google classes. Research has also shown that technology readiness is related to consumer behavior. For example, personality, culture, and social status should be considered as variables when it comes to measuring technology readiness. The ease of use and perceived usefulness of technology are two pivotal factors influencing how individuals adapt to or intend to use it online learning technologies. Other factors that are being researched for more innovative and effective environments of online learning include course design, coordination between online learning platforms and study requirements, skills development, communications channels between students and instructors, assessment and feedback, students’ self-learning, and online learning efficacy.⁵

Several interesting studies have been conducted on the Asian continent. In Malaysia, a study using the Innovation Diffusion Theory (IDT) and Technology Acceptance Model (TAM) model investigated the critical factors that affect students’ preferences for online learning. The study found that students made their decision to use online learning based on perceived compatibility, complexity, and enjoyment.¹⁷ In the UAE, one study found that innovativeness, quality, trust, and knowledge sharing are critical to achieving better online learning system acceptance among students.¹⁸ In KSA, one study highlighted 11 success factors of online learning system implementation in Saudi Arabia, including website quality, technology options, top management support, and online learning awareness by users.¹⁷

Though limited, medical education using online learning is also being investigated by researchers. For example, physiotherapy entry-level education worldwide was highly disrupted by COVID-19 pandemic due to limitations of in-person activities. Utilizing online teaching for introductory physiotherapy courses appeared to be a viable alternative amid the COVID-19 pandemic, proving to be satisfactory for students while maintaining comparable performance levels to traditional face-to-face classes (SOURCE, YEAR). Moreover, the results of initial studies suggest that physiotherapy programs at the introductory level could transition to eLearning to enhance accessibility to higher education. Studies have also suggested that universities should focus on training instructors to cultivate relevant pedagogical skills and provide adequate support in terms of financial, organizational, and technological aspects to enhance education standard to their students.¹⁹ Giacomo’s (YEAR) study summarized digital education in physiotherapy and exploring effectiveness, strengths, weaknesses, and user perspectives to shed light on post-COVID-19 era. This study has carried out quantitative analysis of three types of online classrooms: First, online open-source platforms, which provide consultants and access to knowledge of physiotherapy to students during their clinical practice; second, large online discussions online for case-based learning groups; and third, virtual learning experiences. In brief, digital education has the potential to be utilized in physiotherapy as a substitute resource, such as for theoretical knowledge through online platforms, or as a complementary tool, for instance, in the integration of procedural skills through blended learning, alongside traditional face-to-face teaching methods.

Another example can be found in nursing education. The exploration of Massive Open Online Courses (MOOCs) in nursing education has become a recent focus for researchers.^20,21 These courses have the potential to serve as valuable educational tools for students, particularly as elective options, while at the same time reaching a large audience. The intricate factors influencing their effectiveness indicate the importance of adopting a multidisciplinary approach in both designing and implementing these courses.

For the discussions to follow, it is also worth considering blended learning (BL), which is an educational approach that combines traditional face-to-face instruction with online or digital learning activities. In a BL environment, students typically engage in a mix of activities, such as attending physical classes, participating in online discussions, completing virtual assignments, and accessing digital resources. The integration of technology allows for decrease costs, increased flexibility, personalized learning experiences, and the ability to cater to different learning styles. On the other hand, different studies which have been exploring BL point out the challenges, such as the need for technological system support, the enhancement of teachers’ technological skills in order to create and share teaching materials, the means to ensure transparency in sources and adherence to copyrights, and the requirements on students to independently conduct and manage their academic pursuits.²² In summary, this kind of research into BL is helping decision makers to better understand the online learning potential and structure.

The success of any learning system relies on the user’s usage of the learning system.²³ However, there are main criteria for a successful online learning system, such as the context of an online learning system, student’s acceptance of online learning, assessment of online learning and others. As mentioned above, several studies in the literature have studied different criteria for successful online learning. On the other hand, no study investigates the outcome of an online learning system, which is considered one of the main criteria for the success of online learning systems. Therefore, experimental research is important to identify the efficiency of online learning in order to help decision makers, which is the objective of this research. This study aims to assess the learning outcome of online learning in comparison to on-campus learning in medical higher education.

In most of the research, a thorough analysis of challenges and factors influencing online learning was conducted. These studies are very helpful to address some concerns about online learning, such as understanding students’ needs, choosing a learning system, and developing learning strategies. Despite that, there was no study to estimate the effectiveness of online learning in medical colleges. The purpose of this research is to examine the learning outcomes of an online learning system. Hence, we ask the following questions in that respect: (1) Is online education a successful system for delivering information to medical students? (2) If not, is it a waste of universities’ and students’ time and efforts?

Methodology

This was a randomized experimental study and sample size was calculated using a hypothesis testing method. A total of 203 first- and second-year medical students at King Saud University participated in the study. To make sure that both groups of students heard the identical lecture material from the same instructor at the same time, they were randomly split into two groups. The first group of students attended the lecture in a conventional classroom environment with the instructor physically present. The second group of students electronically listened to the lecture in a different room at the same time. Three days after the lecture, both groups took multiple-choice test of 10 questions. The purpose of the test was to gauge how well they understood the lecture material. The examination consisted of questions spanning the beginning, middle, and end of the lecture. Additionally, one question was included to gauge the participants’ overall comprehension of the entire lecture.

Data Analysis

Statistical Software: Data analysis was performed using the Statistical Package for Social Studies (SPSS 22). The data were described using continuous variables, including means, standard deviations, medians, and modes. For continuous variables that did not exhibit a normal distribution, the Mann–Whitney test was applied. The normality of the data was assessed using the Shapiro–Wilk test. The results of the analysis were considered statistically significant if the p-value was less than 0.05.

Results

Participants Demographics

A total of 203 students from King Saud University’s first- and second-year medical colleges participated in the study (see Table 1). Notably, the study approach used here, which depended on written tests rather than questionnaires, has produced reliable and impartial results. These tests, which evaluated pupils primarily on their retention of the previous lecture topic, were given without prior notice or preparation. With a maximum score of 10, the exam was created to cover every area of the lecture material.

Table 1 Demographic Characteristics of the Participants

The results show that the on-campus group demonstrated significantly higher scores than the online learning group (Figure 1).

Figure 1 Assessment of learning outcomes between on-campus learning and online learning. Chart shows the assessment of learning outcomes between on-campus learning and online learning, where students in on-campus learning scored significantly higher than in online learning, with p-value < 0.001, n=203. Both on-campus and online learning had a mean * SD of 7±1.66 and 5.28±1.94, respectively. Furthermore, the most common methods of getting a degree were 6 and 6, respectively.

Learning Outcomes in Different Group Settings

Correctly interpreting the attainment of learning outcomes of lectures is extremely critical. Therefore, the student’s assessment covered everything that was taught in the lecture. The lecture was 60 minutes long, and the assessment was divided into three sections: the first 20 minutes (start), the second 20 minutes (mid), and the last 20 minutes (end). In this assessment, learning outcomes were measured at three points: the beginning, the middle, and the end. The on-campus students’ group demonstrated similar learning outcomes related to these three points. This indicates that students maintain a similar level of attention and focus during the lecture by maintaining similar scores among the three time points (Figure 2).

Figure 2 Assessment of learning outcomes from on-campus learning. The line chart shows the assessment of learning outcomes from on-campus learning. The mean ± SD of the degree at the beginning, middle, and end were 2.05±0.86, 2.25±0.73, and 2.13±0.81 respectively. The most common degrees at the beginning, middle, and end were 3.2 and 2.

The online students’ group, however, achieved different learning outcomes during the lecture. Students’ scores decreased significantly between the middle and end of the lecture when compared to the beginning and middle. Therefore, this result suggests that students’ attention, concentration, and focus gradually dropped during lecture time, especially at the end of the online group (Figure 3). The comparison of the two groups indicated that, although there were similar learning outcomes at the beginning of the lecture, the learning outcomes in the middle and at the end were significantly lower in the online group than in the on-campus group (Figure 4).

Figure 3 Assessment of online learning. Line chart illustrating the assessment of online learning. The mean ± SD of the degree at the beginning, middle, and end were 1.77±0.95, 1.65±0.92, and 1.35±0.96 respectively. The most frequent degrees at the beginning, middle, and end were 2, 2, and 1, respectively.

Figure 4 Learning Outcomes.

Understanding and Analysis Assessment

During students’ assessment, one deductive question was included, and the right answer was based on understanding and analysis of the whole lecture content. In data analysis, the percentage of students who were able to answer the question was calculated in the two groups. 69.7% of on-campus student groups were able to answer the question compared to 28.5% of online student groups. Statistically, this difference is significant (Figure 5).

Figure 5 Percentage of Student. Chart shows the assessment of learning outcomes from on-campus and online learning. The mean ± SD of the degree at the focus of on-campus and online learning were 0.71±0.45 and 029±0.46, respectively. On-campus learning was focused on 1 and 0, respectively.

Discussion

The findings of this study support the idea that, especially for medical subjects, online learning on its own may not be the best option. For keeping students’ attention and making it easier to impart medical knowledge, on-campus training has proven to be the more efficient way, which might be due to the nature of the medical curriculum. These findings highlight the need to take into account the manner of courses instruction, particularly in fields where a thorough comprehension and retention of difficult subject matter are essential. This study concludes that on-campus learning leads to better outcomes than online learning. Therefore, student engagement and effectiveness methodologies should be enhanced in online learning. Additionally, this finding is based only on theoretical lectures and for practical skills different aspects should be considered. The results have highlighted and emphasized the importance of blended learning as combined learning system technologies and learning processes. It is flexible, enhanced student self-directed, freedom access, cost-effectiveness, and promote communication between students and teachers.

The context of this study also needs to be taken into consideration. This study involved medical students at King Saud University in the Kingdom of Saudi Arabia. Saudi Arabia is a developed nation and a digitally advanced country where online learning can be effective. Economically, all students have access to the Internet and online learning devices, such as phones or university computers.⁶

Our study has also taken into consideration different variable parameters, and some steps have already been taken to prevent any variation in the study samples. All students were at the same location and received the same quality of learning in terms of sound and images. To minimize any differences in content, information, illustrations, and explanations, the online learning group received live broadcast learning videos from the on-campus group. Both groups were assessed at the same time without prior notice to avoid preparation or self-study. Lastly, the experiment and assessment were performed after the COVID-19 pandemic to avoid pandemic-related anxiety that adversely affects student academic performance. We might therefore conclude that student’s academic performance in the online group probably dropped over time due to reduced concentration for learners and lack of teacher focus.

Although online learning is not new to students, COVID-19 is reviving the need to explore it. Generally, the success of an online learning system depends on many factors, such as students’ willingness and their experience with digital technology.¹³ While online learning systems have been implemented in many universities in recent years, the achievement of learning outcomes using these systems have not been thoroughly assessed. As a result, this study investigates learning outcomes as the main measure of online learning.

As noted earlier, to improve the implementation of online learning, much research has been conducted on the challenges and influencing factors. However, less research has been conducted on the quality of online learning systems. It is essential to develop research on teaching and learning techniques as well as the quality of the online learning system in order to make online learning effective. Teaching and learning techniques are a wide subject and are not the focus of this study but are areas for further research suggested by the present study.

Another essential aspect of online learning is the convenience of helpful formative assessments and feedback.⁸ Different assessment methods and examinations are usually used to measure learning outcomes.¹³ This study has used one assessment method, the multiple-choice test, but acknowledges that other methods could be employed in future research.

It is also worth noting that unlike conventional classes, online learning lacks the guarantee of learners’ attendance. Consequently, assessing student engagement and concentration during online learning settings becomes challenging. The readiness of learners for online learning, learners’ self-efficacy, learners’ self-control, learners’ self-directed learning are regarded as prerequisites for improving their academic performance.¹² At the institution used in this study, there was doubt among educators that online learning was not as effective as on-campus learning. In this study, the learning outcomes were measured indirectly by anonymous assessment of students in order to avoid any effect of self-study on the outcomes. To measure attainment of learning outcomes, this study has considered fixing variables, such as time and place, lecture content, and teaching methods.

While the study provides valuable insights into the comparative effectiveness of online and traditional on-campus learning in the context of medical education at King Saud University, several limitations should be considered when interpreting its findings. Firstly, the generalizability of the results may be restricted due to the specific characteristics of the study population and institutional context, potentially limiting their applicability to other education settings with different demographics, resources, and curricula. Along these lines, the study does not account for potential variability in teaching quality or instructional methods across institutes or instructors. Additionally, the focus on short-term learning outcomes assessed these outcomes primarily through written exams and did not fully capture the diverse range of skills and competencies required in medical practice, such as clinical reasoning and communication skills. Despite minimizing any bias by controlling the technological situation for all the students, the study could not control external factors like individual study habits, which could have influenced the results. Finally, although the students were not pre-prepared for the lecture or the assessment, there is the potential influence of the Hawthorne effect, where participants’ awareness of being part of a study may have altered their behavior or performance. This could have also impacted the validity of the findings.

Considerable amount of research has investigated digital health competencies in health care professionals. However, non-validated tools and methodological shortcomings were detected and must be addressed with future research. Furthermore, psychological and emotional elements, along with the ability to self-learn must be investigated more in future research.²⁴ Research on this topic is still at its beginning, where different aspects need to be studied to improve online learning system adoption that can lead universities to better understand their students’ needs and learning techniques and ultimately lead to a successful online learning system.

Conclusion

This study sheds light on the efficacy of online learning in the context of medical education, demonstrating that while it may initially yield similar learning outcomes to traditional on-campus learning, its effectiveness declines as the lecture progresses. Online students showed lower levels of understanding compared to their on-campus counterparts. These findings emphasize the need for educational institutions to thoroughly assess the quality and effectiveness of online learning systems, especially within the context of medical education, to ensure that they meet the educational needs of students. Further research is warranted to explore and enhance online learning methods to better address challenges and provide effective solutions in the ever-evolving landscape of education.

Regardless of general concerns of the risk of increasing screen exposure time for learners,² universities must be up to the task of revolutionary modifications in the higher education system to make higher education more accessible for students and compatible with students and universities in difficult situations. An investigation into the efficiency of online learning, including the long-term impact of online learning, will make education systems better equipped to handle challenges and provide solutions.