Development, Implementation, and Assessment of an Online Modular Telehealth Curriculum for Health Professions Students [Response To Letter]

Dear editor

The authors thank Quereshi et al for their thoughtful summary and review of our study. We wish to offer some contextual insight and practicalities that informed our choices in curricular design and assessment methods.

The writers note that the study is vulnerable to volunteer, also known as selection bias which can affect the external validity of our results. This was due to the MD student arm being composed of students who self-selected to participate in the telehealth curriculum as an elective in their fourth-year capstone course. We acknowledge that mandatory participation from all MD students would strengthen the validity of our results. However, as this was a pilot study being implemented in the context of other courses, this was not an option available to us. This, in our experience, is a common limitation encountered when attempting to implement newly created educational innovations. In our case, this elective experience offered the greatest number of MD students who would be required to complete all the modules. Contrast this to our opportunities with the NP and PA students where all students were required to participate but were only required to complete 2 of the 10 modules. Neither opportunity on its own was ideal for evaluating the curriculum, but together offered the best pragmatic solution for evaluating our full curriculum across multiple health professions. However, the authors believe that by finding minimal differences between the responses of the NP and PA students who were not subject to selection bias and the MD students, it is likely that the effect on the results was minimal. We discussed this challenge of need for flexibility of how our curriculum was integrated into courses stating, “Without this flexibility, it is likely that successful implementation of the curriculum in multiple health professions programs would have been challenging”. We hope that the positive findings of our pilot will support future study of broader, more universal collaborations and applications of our curriculum at both our local institutions, as well as externally, that will address the stated concern.

In response to the concern that the lack of a pre-test survey further weakens our findings, as commented in our discussion, “Pragmatically we were limited to a single touch point with the students.” Additionally, given the voluntary nature of participation in our study, we were concerned that if we inserted a pre-test survey in our online modules, that the response rate to our post-test survey would be adversely affected due to “survey fatigue.” This highlights additional challenges of curricular design and the need to balance ideal design with the practicalities of subject participation. To mitigate the lack of pre-test survey data we asked participants to rate their gain of knowledge in the post-module survey.

Regarding the point raised about our selected assessment measures, “[Lacking] an objective evaluation such as a standardized assessment that could verify whether students’ knowledge and skills improved”, we acknowledge this in our discussion of next steps that “One step is to integrate the modules as part of simulated or actual clinical telehealth experiences along with tools designed to measure student application and performance of telehealth principles. This will allow for more objective measurement of knowledge and skills and application through direct assessment, rather than student self-report”. While it would have been ideal to have objectives that directly measured performance before and after participation in the curriculum, it would have required significant additional funding and resources such as obtaining and funding standardized patients, recruiting and training faculty to observe and assess students, and data analysts to record and process responses. We were appreciative of the level of support we received to conceive, develop, and implement the curriculum as reported. We hope, as stated in our discussion, to have the opportunity to further study the effectiveness of the curriculum using assessments that can detect changes in performance and monitor retention of skills.

In conclusion, we appreciate and agree with Quereshi et al in their discussion of ways to strengthen educational research outcomes. This response hopes to point out the “behind the scenes” context that often interferes with investigators’ abilities to adopt them. When attempting to create innovative curricula or other educational materials, starting with a practical approach that matches available resources and conditions allows the gathering of preliminary data to support further study. We hope to do just that following our initial positive findings.