Examining the Use of Machine Learning Algorithms to Enhance the Pediatric Triaging Approach

Introduction

Machine learning has garnered considerable attention and utilization within the healthcare sector, with the ability to enhance decision-making, improve precision in medical diagnoses, and facilitate the analysis of intricate medical data.^1–5 Machine learning algorithms have the capacity to employ routine healthcare data for predictive purposes, analyze extensive and varied collections of data, and optimize the trade-off between delay and energy in Internet of Medical Things systems.^2,6,7 These algorithms have found applications in diverse fields, including cybersecurity systems, smart cities, healthcare, e-commerce, and agriculture.⁷

In pediatric emergency departments (PED), triage systems play a vital role in effectively prioritizing patients according to the seriousness of their condition. These systems are critical for nurses and healthcare providers in PEDs, as they enable swift identification of children in need of immediate care and those who can afford to wait.^8,9 Conventional emergency triage systems in pediatric care predominantly rely on subjective evaluations and prioritize patients based on limited information. Examples of such systems include the pediatric emergency severity index (ESI), which is susceptible to the healthcare provider’s judgment and exhibits suboptimal discriminatory capability.¹⁰ Another example is the Manchester Triage System, which depends mainly on the presenting signs and symptoms of these patients, without relying on the underlying diagnosis.¹¹ These systems typically lack the ability to perform extensive objective data analysis and utilize predictive modeling, limiting their accuracy and efficacy in identifying patients who require immediate attention or critical care.¹²

Machine learning techniques have the ability to improve systems for prioritizing patient care in PEDs. By utilizing objective data analysis and predictive modeling, machine learning has the potential to enhance the triage methods used to determine the order in which pediatric patients receive care.¹⁰ Machine learning models can generate more precise forecasts of medical outcomes and hospital admission requirements by taking advantage of objective data. By relying on these objective data, machine learning algorithms can guide more accurate hospital admission decision and clinical outcomes.¹³ However, it is essential to acknowledge the challenges in applying machine learning algorithms to the PED triage systems, such as data imbalance, interpretability, and ethical considerations. To address these issues, this study used performance metrics, such as the F1 score and precision-recall curves, which assess model performance beyond simple accuracy and highlight effectiveness in minority classes. Additionally, these algorithms were trained using various demographic and clinical data, minimizing biases in predictions. Furthermore, patient data was anonymized to protect confidentiality, and the study adhered to ethical guidelines.

The current triaging systems lack the ability to perform extensive objective data analysis, which limits their accuracy and efficacy in triaging patients in the PEDs. To address this challenge, this study aims to create a machine learning model that can classify PED patients into three separate urgency groups (non-urgent, urgent, and emergency), which will improve accuracy in pediatric emergency triaging systems. To our knowledge, this is the first study that specifically studies pediatric triage and proposes a comprehensive three-category model that will enhance the accuracy and efficiency of the triage process.

Materials and Methods

This retrospective study did not involve any patient identification information. All patient data were anonymized and utilized solely for the research objectives. In accordance with the principles of the Declaration of Helsinki, King Faisal Specialist Hospital and Research Centre (KFSH&RC) Institutional Review Board granted approval for this study (IRB no. 2190019), ensuring the safeguarding of patient privacy.

The study dataset comprised 38,891 pediatric (age <18) emergency patient records collected from KFSH&RC. Each patient record contained administrative, demographic, and clinical information. We applied several exclusion criteria to filter our patient data. Specifically, we excluded patients who were deceased when they arrived at the hospital. We also excluded patients who were transferred to another medical facility for treatment or admitted to a psychiatric facility. Further, we did not include patients who chose to leave the hospital before their treatment was completed or who elected to leave the emergency department before being evaluated by a physician. By applying these exclusion criteria, we analyzed a cohort of patients who received complete emergency department care and treatment at the hospital facility. We further refined our patient dataset by removing records with missing or clinically implausible data values (Table 1). As a result, the total number of pediatric patient records in the dataset was reduced to 20,317.

Table 1 Accepted Values Range to Be Included in the Study

In the proposed machine learning model, input features were derived from information commonly found in PED triage environments, such as gender, age, mode of arrival, the count of PED visits within the last 72 hours as a proxy for acuity, time of arrival, temperature, systolic blood pressure, diastolic blood pressure, respiratory rate, and pulse rate. Note that neither the oxygen saturation nor patients’ visit purpose records were available in this dataset.

While more granular classifications (eg, five or more urgency levels) could offer additional detail, they were not pursued due to potential challenges in clinical implementation. Increased granularity might lead to decreased interpretability and slower decision-making in high-pressure environments. Thus, the current study pursued the development of a machine learning model that categorizes PED cases into three distinct urgency categories: nonurgent, urgent, and emergency. The three-category system aligns closely with established triage protocols, such as the ESI, making it more practical for integration into existing workflows. This design improves upon previous approaches by providing a balance of accuracy, interpretability, and usability in real-world settings. Additionally, this represents a more expansive and comprehensive typology compared to the simpler two-group classifications examined in most earlier studies. The three-category model developed here aims to capture important subtleties and variations in patient acuity levels that a dichotomous model cannot adequately represent. The dataset used in the study included the ESI provided by nurses on a scale of 1 to 5. To prepare the model’s output, we mapped the ESI levels to specific classifications. Nonurgent cases were categorized as ESI levels 5 or 4, urgent cases as ESI level 3, and emergency cases as ESI levels 2 or 1. Grouping ESI levels into broader categories is intended to minimize the effects of this subjective variability in triage assessments. Categorization helps ensure more consistent outcomes by reducing subtle differences in how nurses might apply the ESI scale. For simplicity, the urgency categories were called A, B, and C, where A = nonurgent (ESI 4–5), B = urgent (ESI 3), and C = emergency (ESI 1–2). To treat data imbalance between different categories, this study utilized the oversampling technique.

To validate the accuracy of the model’s classifications, three independent emergency medicine consultants randomly reviewed and assessed the categorizations provided by the model. Their input confirmed the model’s alignment with clinical expectations, ensuring its reliability and applicability in real-world pediatric emergency settings. Also, it is important to explore and prepare the dataset before training machine learning models. Figure 1 illustrates the relationship and distribution of the numeric input variables. Most of the plots indicating the association between the two variables exhibit a nearly normal bell-curve-shaped distribution of the three outcome categories (A, B, and C). Instances that are not urgent tend to cluster near the center. Furthermore, we examined how each input feature was distributed individually.

Figure 1 Illustration of the binary correlation and distribution plots of the model input features, specifically based on the raw patient records.

Whereas we filtered the features inputted to the model (eg, vital signs), to screen and detect records with incorrect labels, we use confident learning, an automated machine learning technique as it results in more robust and reliable artificial intelligence (AI) models.¹⁴ Figure 2 demonstrates the technique’s objective, which is to estimate a joint distribution between the given or noisy labels and the unknown or uncorrupted labels. This estimation is made under the assumption of a class-conditional classification noise process.¹⁵ In applying confident learning, we used Gaussian Naive Bayes (Gaussian NB) as the backbone machine learning algorithm for its simplicity and effectiveness in handling of normally distributed and differently distributed features. Gaussian NB calculates class probabilities from feature likelihoods.¹⁶ The purpose of using Gaussian Naive Bayes in this study was to identify mislabeled records and clean the dataset.

Figure 2 Confident process is applied to identify and remove mislabeled emergency triage outcomes within the study dataset.

Based on the confident learning step, we eliminated 2080 records, reducing the dataset size to 18,237 records. The records that were eliminated were those with which the Gaussian NB model was not very confident in the triage level it had predicted for them during cross-validation. Given that the model’s predictions for those records were uncertain, they were filtered out. By removing the records with which the model lacked confidence in its predictions, the resulting smaller dataset included only the records where the model predicted triage levels with higher confidence. Also, this study used supervised learning where both the input and output quantities are used in training the machine learning model.

Figure 3 displays the binary correlation and distribution plots of the quantitative input features. These plots depict the relationships among the input features after data preprocessing. Our observations indicated that applying confident learning to clean the dataset resulted in improved information consistency and the elimination of outliers. Bivariate plots showed that there was an overlap, with A overlying B and B overlying C. Additionally, univariate plots demonstrated that the filtered data, both before and after applying confident learning, maintained a similar mean and variance for each of the three categories. However, confident learning tends to filter more instances from classes A and B, which have a more significant overlap. Moreover, the application of confident learning filters tended to limit categories A and B to lower magnitudes in quantities such as temperature, respiratory rate, and hour of arrival. This is reasonable, as higher magnitudes of such vital signs are clinically indicative of more urgent cases under category C. In Figure 4, the dataset was cleaned using confident learning, which involved comparing the quantitative feature distributions before and after the cleaning process. The results of this comparison verify that the feature distributions remained unchanged after applying confident learning to clean the dataset.

Figure 3 The binary correlation and distribution plots of the model’s input characteristics based on the patient records that have been previously processed.

Figure 4 Side-by-side comparison of key triage prediction variables both before (in blue) and after (in Orange) the application of the confident learning algorithm.

We converted categorical data into numeric labels before building the machine learning models. This allowed the models to use those categorical features. The normalization of numeric features was carried out to enhance the efficiency of parameter optimization during training. By reducing oscillation prior to reaching the minimum loss, this normalization technique helps to reduce the time required to find the optimal parameters.¹⁷ This process transforms the cost function into a circular shape in two dimensions and a spherical shape in three dimensions. As a result, the optimizer can converge more quickly, requiring fewer iterations to reach the optimal solution.¹⁸ The input features were normalized and scaled using their respective mean and standard deviation statistics as computed based on the training data split. These statistics were also utilized to similarly normalize the validation and testing data splits.

We evaluated and compared the effectiveness of various types of machine learning classification algorithms for predicting triage levels of patients arriving at the PED. The algorithms employed in the study encompassed a range of techniques, such as regression (eg, logistic regression), instance-based methods (eg, k-nearest neighbors and support vector machines), regularization approaches (eg, ridge classification), tree-based models (eg, decision trees), Bayesian methods (eg, Gaussian NB), dimensionality reduction techniques (eg, linear discriminant analysis and quadratic discriminant analysis), and ensemble algorithms (eg, random forest, extra trees, boosting, Ada Boost, and gradient boosting machines). As the dataset had an unequal distribution of classes, we assessed the models using a variety of metrics to obtain a balanced evaluation. These metrics included accuracy, recall, precision, F-1 score, and quadratic weighted kappa.

Results

The dataset was analyzed, and for the purpose of training and testing, it was divided into two parts: 90% of the records (16,413) were assigned for training, while the remaining 10% (1824) were kept aside for unseen hold-out testing. Before building machine learning model, we examined how each input feature was distributed individually in each of the three triage outcomes (A, B, and C), and we found that for some features the probability was distinguishable. However, for other features, the outcome distributions overlapped and were indistinguishable. This indicates that properly assigning triage outcomes usually relies on using multiple input features together, not features in isolation.

To create a machine learning model with good generalizability to unseen emergency patient data, we utilized 10-fold cross-validation during the training process. 10-fold cross-validation helps ensure that the model’s performance can be extended to data it was not trained on, reducing overfitting. Multiple algorithms were trained, and their 10-fold cross-validation metrics were compared to determine the best-performing model for predicting outcomes for new emergency patient records. Table 2 showcases the results of our experimentation with different machine learning techniques, as each was trained using 10-fold cross-validation. The study showed that ensemble algorithms performed much better than the other techniques across all evaluation metrics, as it has been shown that CatBoost ensemble algorithm achieved the highest F1 scores. Also, we made some adjustments to the settings used to train the CatBoost model. This fine-tuning of the hyperparameters resulted in a slight gain in performance metrics. Specifically, the F1 score increased to 90% which indicates that ensemble algorithms, especially CatBoost, were extremely effective at accurately classifying the data.

Table 2 Comparison of the Effectiveness of Several Machine Learning Techniques

Figure 5 presents the evaluation metrics for each class, while Figure 6 shows the confusion matrix. These metrics and the confusion matrix were calculated on the unseen hold-out test records. The model performed best when distinguishing between non-urgent and emergency cases in the PED. Importantly, the model never incorrectly classified an emergency case as non-urgent, meaning that it did not miss any truly critical cases. Similarly, the model did not incorrectly label any non-urgent cases as emergencies.

Figure 5 Model assessment metrics for the test set for each of the three emergency triage outcomes (precision, recall, and F-1 score).

Figure 6 The confusion matrix produced while testing the trained model on an unknown test set.

Discussion

Pediatric triage sets itself apart from adult triage primarily because of various factors. The foremost distinction arises from the dissimilarities in disease manifestations between pediatric and adult patients.¹⁹ Children possess distinct physiological and developmental characteristics in comparison to adults, which can complicate and prolong the process of diagnosing and triaging certain conditions, such as appendicitis.²⁰ Moreover, pediatric patients who experience traumatic injury, facing an increased risk for mortality due to their physiological differences.²¹ Hence, precise and accurate triage is of paramount importance in the case of pediatric patients. The use of machine learning algorithms to improve pediatric triaging systems is a relatively new area of research. In this study, we developed a machine learning model that categorizes PED cases into three distinct urgency categories: nonurgent, urgent, and emergency.

Machine learning has emerged as a powerful tool in the medical field, revolutionizing biomedical investigations, personalized healthcare, and computer-assisted diagnosis.²² By leveraging advanced algorithms, machine learning can integrate diverse patient data, surpassing human performance to generate accurate predictions.²³ The application of machine learning in healthcare holds the potential to enhance decision-making, refine treatment guidelines, and expedite precise diagnoses, contributing to overall healthcare advancements.²⁴ Within emergency medicine, machine learning has found utility in triage, risk assessment, medical imaging, and optimizing emergency department operations.²⁵ Notably, machine learning algorithms have been employed to anticipate health outcomes for emergency department visits, including fatalities and subsequent emergency service visits, offering valuable insights for timely intervention.²⁶ There are machine learning models that can forecast cardiac arrest during the conveyance of trauma patients, enabling more exact prognostication and potentially saving lives. These models may help identify high-risk patients earlier and guide time-sensitive interventions.²⁷

One of the most promising applications of machine learning is its use in improving emergency department triage systems based on a huge amount of data to prioritize patients. Recent studies have focused on exploring the utility of machine learning algorithms in improving triaging systems in different populations and outcome measures. However, only Goto et al used pediatric data to develop a machine learning model to predict clinical outcomes in PEDs.¹⁰ Although no study has focused on improving PED triaging systems in the pediatric setting, a previous study by Raita et al developed a triaging model that can improve decision-making in the adult emergency department.²⁸ Thus, this study makes a unique contribution by developing a comprehensive three-category triage model in the PED.

In a different context, Goto et al compared critical care outcome predictions, demonstrating machine learning models that exhibited higher discrimination ability (C statistics 0.84–0.85) compared to the reference model (C statistic of 0.78), along with higher sensitivity (0.71–0.78) compared to the reference model’s sensitivity of 0.54.¹⁰ However, the reference model had a higher specificity of 0.91, while the machine learning models had specificities of 0.77–0.86. Similarly, for the prediction of hospitalization outcomes, the machine learning models demonstrated better discrimination ability (C statistics 0.78–0.80) than the reference model (C statistic of 0.73), with sensitivities of 0.67–0.74 compared to the reference model’s sensitivity of 0.83. Regarding specificity, the reference model achieved a specificity of 0.55, while the machine learning models achieved specificities ranging from 0.71 to 0.75.¹⁰

In a comparison of the critical care outcome with the hospitalization outcome, Raita et al showed that the machine learning model outperformed the reference model in all aspects. Both utilize the area under the curve (AUC) as a measure of their discriminatory abilities for critical care outcome; the machine learning models had higher AUC values (ranging from 0.84 to 0.86) compared to the reference model (0.74), indicating better discriminatory ability. They also exhibited higher sensitivity (ranging from 0.75 to 0.86) and lower specificity (ranging from 0.68 to 0.77) compared to the reference model (sensitivity: 0.50, specificity: 0.86). Similarly, regarding the hospitalization outcome, compared to the reference model, the machine learning models had higher AUC values (0.69 vs 0.81–0.82), lower sensitivity (0.87 vs 0.71–0.79), and higher specificity (0.42 vs 0.71–0.76).²⁸ These studies collectively highlight the potential of machine learning techniques in improving triage accuracy and clinical decision-making in emergency care settings.

The present study stands out for its comprehensive approach to pediatric triaging, setting it apart from Goto et al’s study and Raita et al’s study.^10,28 While the latter studies focused on specific outcomes or a comparison between machine learning models and conventional approaches in adult patients, this study took a different approach by developing a three-category triage model that effectively captures subtleties and variations in pediatric patient acuity levels. Moreover, this study used multiple machine learning algorithms, with CatBoost emerging as the top-performing model for predicting triage levels in the emergency department (F1 score of 90%). Hence, this innovative framework has the potential to significantly improve patient outcomes in PEDs.

Limitations

Although this study has several strengths, some limitations should be acknowledged. First, the dataset was obtained from only one center, which may limit the generalizability of the findings to other healthcare settings. Second, the study’s reliance on historical data means that the model’s performance may not account for potential changes in patient profiles or triage practices over time. Third, the dataset used in the study did not provide information on the clinical outcomes of the index visits for the patients.

Conclusion

This study successfully developed a machine learning model for categorizing PED patients into three urgency categories. By considering the unique needs and characteristics of pediatric patients, the model demonstrated the potential to improve triage accuracy and enhance patient care in pediatric emergency departments. While this study’s findings are promising, further research and validation are necessary to ensure the model’s generalizability and effectiveness in diverse pediatric emergency settings. Additionally, future studies focusing on the final patient outcome are needed to improve the direct impact of these tools in the patient outcomes.