Implementing Cancer Pain Management Training for Pharmacists Using the Kirkpatrick Evaluation Model

Introduction

Pain is one of the most common symptoms associated with cancer. A systematic review showed that the prevalence of pain was approximately 33% in treated cancer patients and up to 64% in patients with advanced or metastatic disease.¹ Pain in cancer patients can be caused by the tumor itself or as a result of treatment procedures or treatment-related adverse events.² Cancer pain can significantly affect the quality of life, adherence to treatment, and satisfaction with care for patients and their families or caregivers.^3,4

Inadequate pain management remains a widespread problem worldwide. A retrospective analysis showed that more than 70% of patients reported cancer pain, but even with effective treatment, 50% were still unable to adequately control their pain.⁵ There are many reasons for poor pain control in patients, among which the barriers created by healthcare professionals should not be ignored, such as inappropriate pain assessment and misunderstanding the pharmacology of opioids.^6,7 A survey of healthcare professionals worldwide found that 10.25–13.98% of respondents reported never having received training on opioid for cancer pain, and 46.60% identified lack of pain/palliative care services and lack of knowledge among healthcare professionals as very important potential barriers to cancer care.⁸ In 1986, the World Health Organization (WHO) introduced the three-step analgesic principle for cancer pain management. Guided by this, the National Health Commission of China has made significant efforts to improve the quality of cancer pain management. In 2011, the commission initiated the “Good Pain Management Ward” program and issued the Chinese Guideline for Cancer Pain Management. As a result of these efforts, healthcare professionals have gained a better understanding of the importance of cancer pain management. Nevertheless, China is a country with a large population and a large number of cancer pain patients. In 2022, there were 4.82 million new cancer cases, and more than 2.4 million patients were expected to suffer from cancer pain in China.⁹ China still faces the challenge of a shortage of cancer pain management professionals. A national multicenter survey of Chinese healthcare professionals showed that only 6.3% of the respondents could correctly answer all 10 professional questions about cancer pain, and multidisciplinary collaboration was ineffective in many cases.¹⁰

Effective management of cancer pain requires comprehensive care, including physical, psychological, spiritual, and social support. Collaborative efforts across disciplines are indispensable to address all aspects of the patient’s well-being in the battle against cancer pain. Trained pharmacists collaborate with doctors and play a crucial role in optimizing drug therapy, ensuring patient safety, and improving treatment adherence. It has become standard medical practice worldwide to include pharmacists in the interdisciplinary cancer pain management team.^11–13 However, due to relatively insufficient capabilities and experience, the value of pharmacists in cancer pain management in China is lower than expected. A study conducted in secondary and tertiary public hospitals in China analyzed the ability of physicians, pharmacists, and nurses to practice cancer pain management, their perceptions of it, and their willingness to participate in such management. The results showed that pharmacists scored lowest in all three dimensions compared to physicians and nurses.¹⁴ Replenishing pharmacists with comprehensive knowledge and skills in cancer pain management has become particularly urgent in the face of a large patient population.

Continuing education for healthcare professionals is essential to improve the quality of cancer pain management, and education evaluation is even more critical.^15,16 Evaluation is used to test the effectiveness of education and provide feedback that can improve education. Currently, models commonly used to evaluate training effectiveness include Kaufman’s model, Stufflebeam’s model, Brinkerhoff’s Six-Stage model, CIRO model, and Kirkpatrick’s model. The Kirkpatrick Model, developed by Donald Kirkpatrick in the late 1950s and expanded over time, is a widely used framework for evaluating the effectiveness of training and learning programs.^17–19 The model is structured into four levels, reaction, learning, behavior, and results, each representing a different aspect of the evaluation.²⁰ Kirkpatrick’s model provides a systematic approach to evaluating training programs by considering not only the immediate reactions and learning outcomes but also the practical application and broader effects of the training on individuals and the organization. Each level builds on the previous one, making it a comprehensive framework for assessing the overall effectiveness of training initiatives.

We set out to develop a well-structured cancer pain curriculum for pharmacists in China. This study reports on an in-service training program for pharmacists in cancer pain management developed using Kirkpatrick’s model and evaluates the effectiveness of the training at four levels.

Materials and Methods

Population and Sample Selection

In China, medical institutions are classified into primary (level 1), secondary (level 2), and tertiary (level 3) hospitals. Level 1 hospitals are primary care institutions that provide basic medical services directly to the community. Level 2 hospitals are regional hospitals that provide medical and health services to multiple communities and are responsible for providing operational guidance to and receiving referrals from level 1 hospitals. Level 3 hospitals provide national medical and health services in regions and are medical centers with comprehensive medical, teaching, and research capabilities. Level 2 hospitals are at the top and bottom of the hierarchy in the Chinese health system. With the change in chronic disease prevention and treatment strategies and the implementation of the hierarchical diagnosis and treatment system in China, more and more cancer patients are being referred to secondary hospitals, which have become the most concentrated hospitals for cancer pain patients.

This study was conducted among pharmacists in secondary hospitals in Chongqing, China. These hospitals had implemented the “Good Pain Management Ward” program, equipping with painkillers such as morph and routinely admitting patients with cancer pain. The Chongqing Pharmaceutical Affairs Management Quality Control Center was responsible for recruiting pharmacists. The pharmacists were interested in cancer pain management and were willing to participate in training. There were no restrictions on the age, sex, position, educational background or years of experience in cancer pain management of the participants.

Instructional Design and Delivery

Ten mentors were from tertiary hospitals in Chongqing and included physicians, nurses, and clinical pharmacists. Mentors have senior professional titles and have been involved in cancer pain management and teaching for over ten years. The educational content was designed into six modules with eight teaching units based on cancer pain diagnosis and treatment standards issued by the China National Health Commission and the guidelines for adult cancer pain published by NCCN. These include pathophysiological mechanisms of pain, screening and assessment, pharmacological treatment, identification and management of adverse effects of pain medications, education of cancer pain patients and caregivers, and management of opioids and psychotropic drugs. Each lesson lasted 40–120 minutes, problem-based learning (PBL) was used to provide intensive instruction. All lessons were live classes attended in trainees. The researcher created a WeChat group for each study group with a mentor. Group members could communicate and discuss online after classes while the mentor provided focused advice. Trainees were divided into 10 study groups using the stratified randomization method, with 5–6 trainees in each group, based on their practical experience in cancer pain management.

Evaluation of Training Effectiveness

Level 1: Reaction

This level focuses on participants’ immediate reactions to the training, such as their satisfaction, engagement, and perceived relevance of the training content. The training satisfaction questionnaire was used, comprising two parts. The first part consisted of 9 statements related to course content, teaching approaches, instructional materials, and instructor competency. The second part consisted of 3 statements on clinical competence, ethical and legal practice, and communication skills. Each statement corresponds to 5 ratings: very dissatisfied, dissatisfied, fair, satisfied, and very satisfied. The percentages for each rating were analyzed, and the “satisfied” and “very satisfied” rates were summed. The evaluation was conducted immediately after the training, with questionnaires distributed and collected via mobile electronic devices. The questionnaire was conducted anonymously to ensure the authenticity of feedback.

Level 2: Learning

At this level, the model assesses the actual learning during the training. It examines whether participants gained new knowledge, skills, or competencies. We conducted a comprehensive review of the survey literature on cancer pain management.²¹ An expert panel was established, consisting of two medical oncologists, two medical oncology nurses, two pain physicians, a clinical pharmacist, and a psychologist. After obtaining the necessary permissions, the expert panel carefully selected, adapted, and refined to develop a tool that effectively aligned with the study objectives. The final version of the questionnaire was developed after three rounds of evaluation, discussion, and revision.

The survey was organized into two distinct sections. The first section captured participants’ demographic information. Subsequently, the survey delved into two main areas: pain assessment and familiarity with opioid pharmacology. This section consisted of 13 items, each evaluated using a 4-point Likert scale that ranged from 1 (strongly disagree) to 4 (strongly agree). The questionnaires were administered before and after the training sessions. To prevent unintentional bias in response selection, the distribution and collection of questionnaires was facilitated through mobile electronic devices.

Level 3: Behavior Change

This level addresses whether participants apply what they have learned in their work environment. It measures whether participants use newly acquired skills or knowledge on the job, and whether these changes contribute to improved performance. Four anticipated behavioral outcomes were defined as follows: (1) comprehensive assessment of pain, (2) standardized titration of opioid doses, (3) accurate recognition of opioid-related side effects, and (4) systematic provision of patient education. These behavioral objectives represented different aspects of the behavioral evaluation and carried equal weight, with each contributing 25 points to a cumulative total of 100. Pain assessment scores were assigned as follows: correctly assessing the location, severity, and type of pain, with 3 points for each element; and capturing the patient’s pain relief or exacerbation factors, pain history, pain treatment history, status of vital organ function, psychosocial status, history of psychiatric disorders, history of drug abuse, and history of alcohol dependence, with 2 points for each element. Assessment elements for opioid dose titration include determining whether the patient is opioid-tolerant, correctly converting opioid equivalent doses, determining an appropriate initial dose, using appropriate criteria for dose escalation, and administering an appropriate tapering dose, with 5 points for each element. Elements to assess recognition and management of opioid adverse effects include accurate recognition of and correct response to nausea and vomiting, constipation, dizziness and somnolence, respiratory depression, and delirium, with 5 points for each element. Patient education elements include explaining to patients the causes of cancer pain, emphasizing the importance of pain management, teaching patients the cancer pain assessment tool and how to use it, teaching patients the key points of self-monitoring, and explaining common misconceptions about cancer pain management, with 5 points for each of the above elements.

Three months after training, trainees were recalled to complete a standardized patient-based scenario simulation assessment. The researchers selected a case of small cell lung cancer with rib metastasis and severe pain to write a standardized patient script, including the patient’s attire, the main complaint of pain, and the key points of response when the pharmacist collects medical history, assesses pain, and inquiries about drug adverse reactions. The researchers trained the standardized patient in advance, including the simulation objectives, background knowledge, professional terms, and communication methods, while requiring the standardized patient to strictly follow the script and not to make any changes. The order of evaluation for the trainees was determined by random drawing. The trainees entered the laboratory simulated as an oncology ward in turn, and completed the process of pain assessment, morphine dose titration, identification of morphine adverse reactions, and patient education through interaction with the standardized patient, and then wrote a formatted medical record. The assessment process lasted a total of 90 minutes and was recorded on video. Three experts served as examiners, and scored the trainees by watching the videos and checking the medical records, strictly following the scoring rules. To ensure fairness, the three examiners scored independently, and the average score was taken as the final score for the trainees. Trainees scored ≥15 points in each individual dimension and had a total score of 70–79 as proficient, 80–89 points indicated good performance, and 90–100 points ascribed to excellent achievement. The pass rate was calculated by adding the percentages of trainees classified as proficient, good, and excellent.

Level 4: Results

The highest level of evaluation, this level looks at the broader impact of the training on the organization. It assesses whether the changes in behavior and performance resulting from training lead to tangible outcomes. Six months after the training, the researchers sent an Email to the participants, which linked to an anonymous questionnaire used to evaluate the participants’ roles in cancer pain management practices. The questionnaire covered several areas, such as participation in cancer pain management support activities (eg, pain medication management and dispensation, reporting adverse drug reactions, patient education, and follow-up), a multidisciplinary cancer pain management team, and participation in independent cancer pain consultations or clinics.

Statistical Analysis

Data analysis was performed with SPSS 24.0 software. Quantitative data that exhibited a normal distribution were presented as mean ± standard deviation and categorical data are represented as counts and percentages. Paired t-tests were used to evaluate the scores obtained from the cancer pain management questionnaire before and after training. The correct response rate of the questionnaire was assessed using the chi-square test. A p-value of <0.05 was considered statistically significant.

Results

Fifty-three trainees were enrolled in the study. The demographic characteristics of the participants are presented in Table 1. The specifics of the participants’ previous experience with cancer pain and their training are shown in Table 2.

Table 1 Demographic Characteristics of Participants (n=53)

Table 2 Cancer Pain Management Education and Practice Among Participants (n=53)

The results of the reaction level assessment showed that the cumulative percentage of “satisfied” and “very satisfied” for each of the nine statements exceeded 85%. Furthermore, satisfaction with improving competencies in clinical practice, ethics and law, and interpersonal communication was 94.3%, 84.9%, and 86.8%, respectively, as indicated in Table 3.

Table 3 Participants’ Degree of Satisfaction with the Training (n=53)

The results of the learning level assessment showed that the cumulative correct rate for the 13 items of the Cancer Pain Management Questionnaire was 57.7% before training, which improved to 64.2% after training, with a statistically significant difference (p=0.014), as presented in Table 4. The subsequent stratified analysis revealed that the mean scores for the six specific questions increased significantly after training, as showed in Table 5.

Table 4 Comparison of the Cumulative Correct Rate of the Cancer Pain Management Questionnaire Before and After Training (n=53)

Table 5 Stratified Comparison of Participants’ Scores on the Cancer Pain Management Questionnaire Before and After Training (n=53)

The findings of the behavioral level assessment indicated that the average score for the four behavioral benchmarks exceeded 15 for each of the 53 trainees. The pass rate for the trainees was 86.8%. The percentages of trainees who achieved proficient, good, and excellent ratings were 18.9% (10/53), 50.9% (27/53), and 17.0% (9/53), respectively.

For the 6-month follow-up assessment, 53 questionnaires were distributed, and 49 were collected, with a response rate of 92.5%. The results demonstrated that all respondents participated in cancer pain management practices. Of these participants, 85.7% (42/49) were part of the multidisciplinary cancer pain management team, and 53.1% (26/49) conducted cancer pain consultations or attended outpatient clinics.

Discussion

The core principle of the Kirkpatrick model emphasizes the concept of “beginning with the end in mind”, where the desired result serves as the initial foundation for designing training programs.²⁰ Ensuring that the training content is in line with the trainees’ requirements is imperative to improve trainee satisfaction. This requires an initial evaluation of the trainees, establishing a baseline. Our baseline survey results indicated a widespread lack of knowledge and insufficient training in cancer pain management within the pharmacist employed in secondary hospitals. This underscored the need for customized training interventions.

Within the multidisciplinary cancer pain management team, the pharmacist plays a pivotal role in optimizing the efficacy of drug therapies and safeguarding patient medication safety.²² The curriculum designed for pharmacists should prioritize several vital areas, including pain pharmacotherapy, the ability to recognize and manage adverse effects of analgesics, proficient pain screening and assessment, as well as the education of patients and caregivers. Constructing an effective curriculum is only half the battle; choosing appropriate teaching methods is equally vital for enhancing learning. This study used a combination of learning groups and PBL techniques. This strategic approach required trainees to adopt an active position as learners and creative problem solvers, in contrast to the passive recipients of information. This dynamic approach increased trainee motivation and instilled a sense of achievement upon task completion.²³ The participants’ overall satisfaction with the training reached an impressive 88.7%. Furthermore, the rate of correct responses to the cancer pain management questions increased by 6.5% after training, indicating a discernible improvement in learning outcomes. This positive change means that training successfully addresses reactive and learning-focused objectives.

The core objective of the training is to equip individuals with the necessary knowledge and skills to perform their job tasks proficiently, translating the acquired knowledge into practical application. However, researchers have highlighted the gap between levels 2 and 3 of the Kirkpatrick model, emphasizing the challenge of connecting the acquisition of new skills or knowledge (level 2) with actual behavioral changes in the workplace (level 3).²⁴ Fostering this bridge demands a trainee’s active participation and belief in their capacity. Factors influencing a trainee’s readiness include their motivation to engage, the pertinence of training content to their job, and the supervisors’ attitudes toward the training. This study addressed these aspects by involving volunteers in a standardized cancer pain management unit at their hospital. In addition to willingness, cultivating confidence in taking action is imperative. Crafting a post-training action plan is one of the most effective methods to boost this confidence. This research identified four predefined behavioral targets and communicated the assessment date to trainees. The researcher established these predetermined benchmarks to act as catalysts, motivating the trainees to apply their newly acquired knowledge in practical scenarios. Beyond these downward drivers, trainees often require upward drivers to elicit the desired behavioral transformation. Examples of such upward drivers include mentorship, recognition through awards, and the presence of role models. In this study, the researcher established a WeChat group that features mentors for study participants. This platform facilitated seamless communication, reciprocal learning, and continuous guidance and support within the study group. This mentoring structure was implemented to encourage and support the desired behavioral change. The results of this study revealed the effectiveness of this approach. The behavioral assessment of the trainees produced an 86.8% pass rate. Once these pivotal behavioral changes manifest, achieving the ultimate objectives becomes more feasible. In particular, the trainees were actively involved in varying levels of cancer pain management within six months after training, illustrating the effectiveness of the training event at the outcome level.

Limitations

Conducting behavioral assessments poses more significant challenges compared to reaction and learning evaluations. Kirkpatrick identified three reasons contributing to these challenges. First, participants need the opportunity to make behavioral changes. Second, the timing of behavior change remains unpredictable. Last, the influence of the organizational climate on behavior change is inherently uncertain. Therefore, it is advisable to perform evaluations repeatedly to determine the accurate outcomes of behavioral modifications. In the current study, limitations hindered the repetition of behavioral evaluations due to inherent implementation difficulties. Furthermore, the absence of a control group in this study did not allow for the exclusion of Hawthorne or Rosenthal effects or a thorough analysis of other factors that may have influenced the results.These limitations require ongoing research to produce more comprehensive and nuanced data.

Conclusion

In summary, the results suggest that the training program significantly improved pharmacist understanding and skills in cancer pain management and demonstrated its effectiveness at all four levels of the Kirkpatrick model. This study offers valuable information on creating effective training programs for cancer pain management.