Back to Journals » Risk Management and Healthcare Policy » Volume 18

Multicriteria Risk Evaluation Model: Utilizing Fuzzy Logic for Improved Transparency and Quality of Risk Evaluation in Healthcare

Authors Drnovšek R , Milavec Kapun M , Šteblaj S, Rajkovič U 

Received 8 August 2024

Accepted for publication 1 January 2025

Published 26 February 2025 Volume 2025:18 Pages 637—653

DOI https://doi.org/10.2147/RMHP.S490598

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Jongwha Chang



Rok Drnovšek,1,2 Marija Milavec Kapun,3 Simona Šteblaj,1 Uroš Rajkovič2

1University Medical Centre Ljubljana, Ljubljana, Slovenia; 2University of Maribor, Faculty of Organizational Sciences, Kranj, Slovenia; 3University of Ljubljana, Faculty of Health Sciences, Ljubljana, Slovenia

Correspondence: Rok Drnovšek, University Medical Centre Ljubljana, Zaloška cesta 2, Ljubljana, SI-1000, Slovenia, Email [email protected]

Introduction: Risk management is essential for quality assurance in modern healthcare organizations. Risk matrices are widely used to evaluate risks in healthcare settings; however, this approach has noteworthy weaknesses and limitations. This paper introduces a novel risk evaluation model that utilizes multicriteria decision-making and fuzzy logic, to enhance the transparency and quality of the risk evaluation process in healthcare.
Methods: The Multicriteria Evaluation Model was developed using the Decision Expert method and expert knowledge integration. Fuzzy logic was integrated within the model, using partial degrees of membership and probabilistic analysis, to address uncertainties inherent to healthcare risk evaluation. The evaluation model was tested with healthcare professionals active in the field of risk management in clinical practice and compared with the risk matrix.
Results: The designed evaluation model utilizes multicriteria decision-making while encompassing the risk matrix framework to boost user understanding and enable meaningful comparison of results. Compared with the risk matrix, the model provided similar or marginally higher risk-level evaluations. The use of degrees of membership enables evaluators to articulate a wide range of plausible risk consequences, which are often overlooked or ambiguously addressed in the traditional risk matrix approach.
Discussion and Conclusions: The evaluation model demonstrates increased transparency of the decision-making process and facilitates in-depth analysis of the evaluation results. The utilization of degrees of membership revealed distinct strategies for handling uncertainty among participants, highlighting the weaknesses of using single value evaluation approach for the presented and similar decision problems. The presented approach is not limited to healthcare-related risk evaluation, but has the capacity to improve risk evaluation practices in diverse settings.

Keywords: multi-criteria decision-making, risk management, fuzzy logic, decision support, patient care, DEX

Introduction

The report by the Institute of Medicine (US) Committee on Quality of Health Care in America, titled “To Err is Human: Building a Safer Health System”, underscores the critical role of healthcare quality assurance in preventing unnecessary patient deaths within healthcare organizations. Using data extrapolation, this report identified medical errors as a significant contributor to mortality, surpassing more commonly recognized causes such as vehicular accidents, breast cancer, and AIDS.1 Similarly, a later dated paper, summarizing findings from five studies on preventable medical error-related deaths in the United States reports a substantial impact of medical errors on mortality rates, estimating that approximately 251,454 annual patient deaths can be attributed to preventable fatal adverse events.2 Although, some researchers have raised concerns about the accuracy of these estimates,3 up to date empirical evidence still highlights the substantial impact of avoidable patient harm due to different types of adverse events.4 The most recent study reviewed estimates that nearly 800,000 patients in the United States die or suffer serious harm annually as a result of diagnostic errors alone.5

In recognition of these findings, continuous quality assurance and quality management efforts have become vital components of contemporary healthcare practices.6,7 International organizations that provide accreditations for healthcare organizations play an important role in healthcare quality management and are increasingly prevalent today.8 Accreditation procedures and standards by the International Standards Organization (ISO) and the American Accreditation Commission International (AACI) are used internationally and are also predominately present in our region. They serve as an essential quality assurance framework for healthcare organizations and incorporate risk-based thinking as an integral strategy for quality assurance, solidifying risk management as a critical approach for quality improvement.9–11 According to ISO, risk is defined as the effect of uncertainty on objectives, expressed as a combination of risk sources, potential events, likelihood, and consequences. The framework outlines risk assessment as a process of identifying, analyzing, and evaluating risks in order to achieve the best risk treatment solutions.11,12 The focus of this paper is risk evaluation practices, which are centered on comparing risk analysis results to predetermined risk evaluation criteria, aiming to support organizational decision-making regarding risks and appropriate mitigation responses.11 The purpose of risk evaluation is to assess and compare risks, to determine the effectiveness of existing controls, or the need for additional measures.13

Risk matrix is a widely used approach for evaluating the risk levels of individual risks in healthcare settings. It is a graphical representation of risk, determined as a combination of likelihood and severity criteria.13 The level of risk is calculated as the product of these two criteria, with various approaches of defining, categorizing, and interconnecting its components.13–15 The labeling of criteria can vary in different sources, including terms for risk consequence evaluation such as impact, severity, or consequence, and risk likelihood evaluation terms such as probability, likelihood level, or likelihood. In our paper, the terms “Risk level”, “Likelihood”, and “Severity” were used as Risk level = Likelihood × Severity.

Initially introduced in aviation,16 the risk matrix has since been widely adopted in various high-risk industries, including healthcare, nuclear power generation, the oil industry, and manufacturing.13 It is broadly applicable in various contexts17,18 and is commonly used to evaluate risk in health and healthcare settings.13,19–22 However, it lacks a strong research basis and empirical support.23,24

Some authors have criticized the risk matrix approach, highlighting issues such as subjectivity, poor transparency, ambiguity of inputs, misleading risk classification, and inadequate handling of uncertainties that need to be addressed.15,23,25,26 Consequently, the results of the evaluation process may be misleading, inaccurate, and lack transparency. Inadequate transparency of the decision process hinders the in-depth interpretation, justification, and comparison of risk evaluation results, making this approach best suited for cautious and highly customized applications.23,27 Although the risk matrix is the most prevalent approach for risk evaluation in healthcare, methods such as Failure Mode and Effects Analysis (FMEA) and Healthcare FMEA are also commonly used.28,29 These, however, suffer from similar weaknesses regarding poor objectivity and inadequate transparency of the evaluation process. Integrating multi criteria decision-making methods could improve the quality of risk evaluation practices in healthcare.30 Furthermore, advanced risk analysis approaches, including Event Tree Analysis, Fault Tree Analysis, and Bowtie diagrams, have also been present in contemporary healthcare risk management practices.31 These approaches highlight the importance of acknowledging uncertainties in healthcare risk evaluation, but lack a clear and transparent mechanism for transitioning these uncertainties into the risk evaluation process.

This paper aims to introduce the results of the initial clinical testing and conceptual validation of The Multicriteria Risk Evaluation Model for healthcare risk evaluation, which is based on the risk matrix approach and fuzzy logic integration. The presented evaluation model aims to address some weaknesses of the risk matrix approach while maintaining adequate applicability and enhancing the risk evaluation quality within a hospital setting. Recently published reviews highlight the wide applicability of multicriteria decision-making methods in addressing complex decision problems in healthcare and other fields, including their use in risk evaluation and management.30,32 Additionally, these methods can incorporate fuzzy logic into the risk evaluation framework, to assist decision-makers in managing uncertainties,29,33 demonstrating the feasibility of this approach for improving uncertainty handling in healthcare and other high-stakes environments.

The novelty of the presented evaluation model lies in its ability to address some of the weaknesses of the risk matrix approach while maintaining strong intuitiveness and adequate applicability for evaluating diverse risks within a hospital setting. Our primary objective is to improve the transparency of the decision-making process and support evaluators in holistic risk evaluation. Additionally, we recognize and address uncertainties related to risk severity evaluation arising from the wide variety of potential outcomes associated with individual events in healthcare. In-depth risk analysis involves the use of clinical data and expert knowledge to uncover various uncertainties regarding a range of potential risk-related outcomes in healthcare settings. The presented model differs from the commonly used fuzzification of linguistic expressions, as it uses partial degrees of membership as evaluators’ inputs instead of crisp values. Its purpose is to empower decision makers to recognize, utilize and address risk evaluation-related uncertainties, which, when using the risk matrix, may otherwise be handled ambiguously with less precision and lack of transparency.

Methods

The Multicriteria Risk Evaluation Model was developed with scientific and expert knowledge integration using the Decision Expert (DEX) method. Fuzzy logic was integrated into the evaluation model to address uncertainties in the risk evaluation process. The proposed model was developed and tested at the Slovenian University Medical Centre. Approval from the ethical committee of the University of Maribor, Faculty of Organizational Sciences, was obtained prior to conducting this study.

Model Development

Accreditation guidelines mandate the maintenance of risk registries by healthcare organizations to structure and enhance their risk management practices.9,11 An analysis of risk registry entries from the past five years within the participating clinic was conducted to identify the various types of risks present in healthcare organizations. The findings of the risk registry analysis guided the selection and arrangement of evaluation model criteria.

Further development of the model consisted of integrating expert knowledge, scientific findings in the field of healthcare risk management, and established decision support methods. A multicriteria decision support model was developed. Four unique variations of the evaluation model were designed during the development process. Each variation of the evaluation model was presented and tested within the expert group to obtain suggestions for improvement and to advance professional and structural adequacy. Following this, the criteria selection, arrangement, and their individual contributions to the final level of risk were consolidated to form the presented Multicriteria Risk Evaluation Model. The participating expert group consisted of three registered nurses and one doctor of medicine active in the field of risk management (personnel responsible for risk management in the participating clinic) and three experts active in the field of operational research and decision support (previously engaged in integrating decision support tools for healthcare-related decision-making).

DEX Method

The Multicriteria Risk Evaluation Model was developed using the DEX method. The DEX method uses qualitative multi-attribute utility theory, in which the resolution of multiple small problems enables the resolution of a more complex decision problem. In DEX decision models, the set of attributes or criteria is denoted by X = {x1, x2, …, xn}. These criteria can be represented in a hierarchical structure resembling a tree (tree of attributes) with aggregated criteria (hierarchically higher) and basic criteria (hierarchically lowest). The final step of the decision process (in our case, the final evaluation of risk) is represented by the hierarchically highest criterion.34

The DEX method differs from most conventional multi-attribute decision methods because it uses qualitative rather than quantitative criteria. Alternatives are evaluated using the domain value (Di) of criterion xi, where Di is a finite set of at least two discrete values Di = {di1, di2, …, dij} arranged orderly from least (di1) to most (dij) desirable according to the decision problem. A decision model is used to evaluate alternatives. Alternatives are an infinite set of options (in our case risks). Each alternative is evaluated by selecting the most suitable option from the domain value for all basic criteria. The values for the higher-level aggregated criteria are derived using predefined rule-based utility functions through simple if-then decision rules rather than quantitative weights or mathematical utility functions.

In contrast to the usual weighting sum models, the DEX method enables the weights of the criteria to be dependent on their values. For example, a criterion may have little or no influence on the evaluation of a hierarchically higher criterion when its value is moderate or low. However, the same criterion could have a significant or even decisive impact if its value is high.34 DEX method was chosen for its relevance for the particular decision problem, our familiarity with the approach and its proven success in previous healthcare-related applications.35 For the development of the model and evaluation of alternatives, the freeware DexiSuite – DexiWin version 1.2 was used, along with the software’s built-in tools for integrating fuzzy logic.36

Uncertainty Modeling

In The Multicriteria Risk Evaluation Model, fuzzy logic and probabilistic analysis were employed to address the uncertainty regarding the multitude of possible risk outcomes.

The key aspect of fuzzy logic comprises the fuzzy sets introduced by Zadeh in 1965. The author presented the definition of fuzzy sets and methods for operating with them. In contrast to classical sets, where elements are attributed clear membership, the elements of fuzzy sets are ascribed a degree of membership.37 Therefore, a fuzzy set is defined by a membership function within the interval [0,1]. If X is the universe of discourse with individual elements x and A is a fuzzy set, this set is defined by a membership function that transforms each element of set X into a degree of membership within the interval from 0 to 1.38 The concept of fuzzy logic aims to avoid strict categorization of properties into one of two values: complete membership or complete non-membership. Instead, fuzzy sets allow partial membership in the set.39

Guidelines by Hawer40 were used to determine the most suitable approach for applying fuzzy sets in our decision problem. To express evaluation uncertainty regarding a multitude of possible risk outcomes, degrees of membership expressing discrete probabilities and probabilistic analysis of inputs were used. The used DexiSuite – DexiWin36 facilitates the use of partial degrees of membership and probabilistic analysis, which were used in the model design.

Evaluation Model Testing

Personnel active in risk evaluation at the department for quality assurance of the participating clinic participated in the testing of the model. To ensure maximum understanding of the evaluation model by the participants, interactions with the model took place in individual interviews and in one case with two participants simultaneously. Interactions with participants were audio-recorded to support future analysis of the findings and gather impressions regarding the usage of the evaluation model.

Preliminary to testing, a selection of risks evaluated in this study was predetermined. This selection included the risk of patient fall, risk of pressure ulcer, risk for employee burnout, and risk of workplace injury. These risks were selected because they were commonly recognized in diverse clinical environments within the organizational risk registry. The participants were asked to evaluate at least one clinical risk of their choice. Data regarding the incidence of the four mandatory risks were provided to ensure a relevant comparison of the risk evaluation results among the participants. For the risks of their own choice, participants were asked to use their knowledge of incidence to assess the likelihood of risks.

The testing involved two meetings spaced one to four days apart (depending on the participant’s availability), each lasting 75–100 minutes. The meetings were held from September 19, 2023, to January 25, 2024.

In the initial meeting, individual participants first evaluated all risks using the contemporary organizational risk matrix, as succinctly presented in Figure 1. It employs a 4×4 grid to combine “Severity” and “Likelihood” criteria providing ordinal nine-point scale risk-level evaluations.

Figure 1 Contemporary risk matrix used to evaluate risk in the participating organization.

This was followed by a presentation of the structure and functionalities of The Multicriteria Risk Evaluation Model and a demonstration of its use in hypothetical examples. The participants were first introduced to simple decision models to demonstrate the features of multicriteria decision-making. Next, a summary of weights for individual aggregated criteria was presented, and participants were asked to predict the values of the aggregated criteria based on the provided weights. This was done to provide insights into the workings of the evaluation model and to gather feedback on participants’ understanding of how evaluations of basic criteria influence the values of hierarchically higher aggregated criteria. An example evaluation was then conducted by the researcher, who explained the evaluation process in detail. Each participant then carried out an evaluation of another example risk under the supervision and guidance of the researcher, before proceeding with evaluations for data collection. Participants then engaged with the model using a structured form to evaluate the risks according to the basic criteria of the model using single value evaluation approach. In the second meeting, the contents of previous meeting were summarized. The participants were then familiarized with the concept of fuzzy logic and the use of degrees of membership to express discrete probabilities in consequences evaluation before applying the approach to reevaluate the “Severity” criterion of the analyzed risks. Similarly as in the first meeting, the researcher demonstrated an evaluation of risk using fuzzy logic, which was followed by an individual participant evaluating another example of risk using fuzzy logic under supervision and with additional guidance, if needed. During both meetings, the risks for blood transfusion reactions and the risk of inadequate patient privacy assurance were used as examples.

Participants were not informed about the final evaluations of The Multicriteria Risk Evaluation Model during testing and were instructed not to base their individual evaluations of the criteria on previous evaluations of the same or similar risks.

Results

The Multicriteria Risk Evaluation Model was structured in accordance with the risk matrix framework to aid evaluators in understanding the proposed approach. The final version of the developed model consists of ten basic and six aggregated criteria. Table 1 represents the hierarchical structure of The Multicriteria Risk Evaluation Model, with the assigned descriptions of the individual criterion.

Table 1 Evaluation Model’s Criteria and Their Descriptions

The presented risk evaluation model was designed using Slovenian language. The parts featured in this paper were translated separately by two researchers and were uniformed with consensus.

Domain Values and Utility Functions Determination

Similar to the selection of criteria, the qualitative domain values selection acknowledged contemporary risk evaluation methods while aiming for increased objectivity of inputs. As an example, we present the domain values of the criterion “Estimated occurrence” in Table 2.

Table 2 Domain Values and Their Descriptions of the Criteria “Estimated occurrence”

The values are arranged from most to least desirable, where a more desirable value results in a lower, and a less desirable value results in a higher evaluation of risk. Similarly, the domain values of the remaining criteria were determined. To evaluate risks using the model, the evaluator selects the most appropriate value for a particular risk, while the values of hierarchically higher aggregated criteria are derived by a predetermined utility function consisting of individual simple if-then rules.

The DEX method is particularly suitable for precise modelling of expert knowledge, as rules in the evaluation model can be individually assigned. Leveraging the functionality of this method allows for the contribution of the criterion’s impact, based on its unique value. Through iterations, decision rules were determined within the expert group. The functionality of individual rule determination was most profoundly applied in the criterion “Number of affected”, where traditional sum weights were unable to produce a satisfactory outcome. Decision rules for this criterion were determined so that the criterion value had minimal impact on the evaluation when the lowest “Individual” value was selected. However, the expert group recognized that higher values for the criterion demanded a significant shift. The evaluation model, therefore, strongly favors higher grades for the criterion, to the extent that it becomes challenging for the model to derive a desirable overall evaluation of risk when the high value for the criterion is selected. This approach was adopted based on the decision that even minor consequences should receive a high evaluation of consequences when the event affects a larger number of stakeholders.

To present an example of a utility function, the aggregated criterion “Operational capability” was chosen. Determined decision rules of the utility function are given in Table 3. Additionally, we provide a graphical representation of the utility function in Figure 2, generated with the used software, DexiSuite. The figure illustrates all possible evaluations of basic criteria “Work processes” and “Work relations” with resulting evaluation of the aggregated criterion “Operational capability”.

Table 3 Specific Rules Defining the Utility Function of the Criterion “Operational capability”

Figure 2 Representation of utility function of the criterion “Operational capability”.

Similarly, individual decision rules for the remaining aggregated criteria were determined through collaboration among the members of the interprofessional expert group.

Model Testing Results

All personnel active in risk evaluation at the Department for Quality Assurance of the participating clinic were invited to partake in the testing of the model, to which seven participants (five nurses, one doctor of medicine, and one physiotherapist) from diverse clinical environments within the participating clinic agreed to participate. Among the seven participants, 34 risks were evaluated using three diverse risk evaluation methods: the risk matrix, The Multicriteria Risk Evaluation Model single value evaluation, and The Multicriteria Risk Evaluation Model with partial degrees of membership. The testing yielded 102 risk evaluations included in our analysis. In the study, evaluations of nine unique risks were done. This included four mandatory preliminarily delegated risks with the given incidence data and five risks of the participant’s own choice, for which likelihood estimations were based on expert knowledge. Two participants chose identical risks to evaluate, whereas one declined the evaluation of the additional risk of their own choice.

Similar to the organization’s 4×4 risk matrix, the evaluation model uses a nine-point scale to rank risks. This decision was made to enhance user acceptance, improve comprehension of the risk evaluation process, and enable meaningful comparison between the two approaches. Additionally, The Multicriteria Risk Evaluation Model‘s nine-point ordinal scale was binned into classes, grouping risk evaluations into “Low risk” (1, 2), “Moderate risk” (3, 4), “High risk” (5, 6), and “Critical risk” (7, 8, 9), following the structure of the existing risk matrix approach as presented in Figure 1.

Both the evaluation model and the risk matrix yielded an equivalent number of diverse evaluations, categorizing risks into six distinct levels across all above presented risk classes (ranging from “Low” to “Critical”). In both evaluation approaches, no significant evidence was found to support the impact of individual risks on the final evaluation. Some differences in the variance patterns were observed between the approaches, which could indicate a difference in the consistency of evaluations, with the evaluation model showing slightly lower levels of variance across all four mandatory risks. The distributions of evaluations for both analyzed approaches are presented in Figure 3.

Figure 3 Distribution of final risk evaluations by both approaches.

The distributions of final risk evaluations for both analyzed approaches are shown as violin plots, with black dots representing the individual evaluations of all seven participants. Risks are given at the top of each panel. Only the four mandatory risks were included into this section of analysis, as all participants were asked to evaluate these risks, while the fifth risk was chosen individually by each participant. Similar to the risk evaluations distribution analysis, no significant correlations supporting interrater reliability between evaluators were observed in the data.

A preliminary sensitivity analysis was conducted to evaluate the model’s performance. A “±1” sensitivity analysis was used to assess the impact of changing individual values of the basic criteria on the final risk evaluation. This approach examines how increasing or decreasing the evaluation of an individual basic criterion by one rank affects the overall risk evaluation.

The results of the sensitivity analysis for an alternative “Risk of patient fall” evaluation, as evaluated by a participant, are presented in Table 4. The risk was evaluated as “Moderate risk” by the participant. The sensitivity analysis demonstrates that the criterion “Estimated occurrence” significantly impacts the final risk evaluation, raising it to “High risk” if increased by one rank. Additionally, the criteria “Health” and “Satisfaction” decrease the final evaluation to “Low risk” if their values are lowered by one rank.

Table 4 Results of ±1 Sensitivity Analysis for the Example Alternative “Risk of patient fall

The evaluations of the same risks by the same evaluators using both approaches differed among the approaches, with the evaluation model use resulting in slightly higher risk evaluations. This was verified by the Wilcoxon signed-rank test for related samples comparing nine-point scale evaluations from both tools (p = 0.040) but was not statistically significant when comparing data binned into classes (p = 0.075). Discrepancies in the assigned classes for the evaluated risks are shown in Figure 4.

Figure 4 Discrepancies in final risk evaluation - risk level by the evaluation model subtracted by the risk matrix risk level evaluation.

The distribution of the evaluation discrepancies shown in the graph exhibits slightly positive skewness. The mode of evaluation discrepancies was zero, indicating that, most commonly, there is no difference in the risk class assignment between the two approaches. However, The Multicriteria Risk Evaluation Model tended to assign higher risk evaluations more frequently than lower ones compared to the risk matrix approach.

To demonstrate the reasoning for these discrepancies, two representative cases of individual participants’ evaluations were selected: a case of risk for patient fall being evaluated with high consistency, and a case of risk for employee burnout with a low level of consistency using all three evaluation methods. For the example of patient fall risk, the participant evaluated the risk as a “Moderate risk” (“Likelihood” = 2 and “Severity” = 2), with both the risk matrix approach and the tested model. The second example of the risk of employee burnout demonstrates a greater discrepancy between the approaches. The evaluator first evaluated the risk of employee burnout as a “Low risk” (“Likelihood” = 1 and “Severity” = 2) using the risk matrix. However, using The Multicriteria Risk Evaluation Model, the same evaluator assessed the risk as a “High risk” (“Likelihood = 2 and Severity = 3). Both evaluations using the model are shown in Figure 5a.

Figure 5 Outputs of the evaluation model representing: (a) The final evaluation and evaluation of the two main aggregated criteria of both presented risks, (b) The evaluation of the aggregated criterion “Likelihood” of the presented risk for employee burnout (c) The evaluation of the aggregated criterion “Severity” of the presented risk for employee burnout.

To outline the reasoning for the presented discrepancy, in this section, we address the higher evaluation of the risk likelihood. The participant evaluated “Estimated occurrence” as “Low” (estimating about annual occurrence of risk) using the evaluation model. This corresponds to provided data that stated from approximately two to five annual occurrences in the previous five years, and indicates a decreasing trend. The evaluator recognized a decreasing trend in the risk occurrence over the five-year period and deemed this occurrence as expected and acceptable given the nature of the risk and capabilities of the healthcare organization. This resulted in the evaluation of “Likelihood” = 2. The evaluation of all of the above-mentioned criteria is presented in Figure 5b.

On the other hand, using the risk matrix, the evaluator evaluated the risk likelihood as “1” representing the occurrence as “I do not believe that the risk could occur in the near future.” This evaluation is less expected for events (according to the presented data) that commonly occurred in previous years. Therefore, the particular participant was asked why they had assigned this specific grade when the evaluations were complete. The participant explained that the event is likely to occur in the near future, but the evaluation was chosen to depict the difference in likelihood in comparison to previously evaluated risks (“Likelihood” criterion evaluated as 2), that in preceding years occurred approximately 30 to 70 times annually (Risk of patient fall and risk of pressure ulcer). This occurrence of surprisingly low gradings of “Likelihood” criterion using the risk matrix was not isolated to a single evaluator. In fact, the likelihood rating of “1” was unexpectedly assigned in five instances using the risk matrix approach but only appeared in the evaluations of the last two risks according to their order of evaluation. Conversely, the evaluation model did not yield an unexpectedly low likelihood evaluation.

To further analyze the discrepancy in results, we now address the difference in the criterion “Severity” (evaluated higher in comparison to the risk matrix). The higher evaluation could be attributed to the contribution of the simultaneous consequences of employee burnout identified in different organizational aspects. The evaluation of these criteria is shown in Figure 5c.

The evaluator recognized the impact of employee burnout not only in the criterion “Stakeholder wellbeing”, but also in criterion “Corporate standing” and criterion “Operational capability”. The impact of risks on all these criteria increased the evaluation of criterion “Severity” to 3. However, incorporating probabilistic analysis into the evaluation process revealed additional differences in evaluations using the analyzed approaches. To further investigate the discrepancies in “Severity” evaluations, we present the findings of the degrees of membership and probabilistic analysis use in risk severity evaluations.

The participants were offered an option to use degrees of membership for risk consequence evaluation. Fuzzy logic broadens the scope of conventional binary logic by embracing the notion of partial truths, which lie between the absolute values of ”true” and ”false.” This is more closely related to human thought processes and can facilitate uncertainty integration in the decision-making process, making it particularly applicable in complex decision-making environments where clear-cut answers are rare. Using this approach, The Multicriteria Risk Evaluation Model does not provide crisp results. Instead, decision makers input their fuzzy evaluation of basic criteria and are also presented with results in varying degrees of membership, reflecting uncertainty and depicting the nuanced nature of the risk evaluation process.

In some cases, the fuzzy evaluation was closely related to the single value method evaluation, attributing the highest degree of membership to the same level of risk. To continue with our representative examples, this is true in the case of patient fall risk, where the highest degree of membership (0.57) was assigned to the same class of “Moderate risk”, while adjacent classes were assigned lower degrees of membership (“Low risk” – 0.20 and “High risk” – 0.23). This was not the case for employee burnout evaluations. When given the option of using the degree of membership, the evaluation of risk shifted. While the participant evaluated the risk of employee burnout using single value as a “High risk”, the evaluation using degrees of membership derived an evaluation of “High risk” with the membership of 0.48, but importantly the evaluation of “Moderate risk” with the degree of membership 0.52. The final evaluations of both risks using probabilistic analysis and compared with their evaluations using a single value method are presented with the program output in Figure 6.

Figure 6 Output of the valuation model representing final levels of risks evaluated, using single value and degrees of membership.

The different degrees of membership are a result of a variance in evaluation of “Severity” criterion, which reflects different possible scenarios that the participant anticipates. In both cases, the evaluation of “Stakeholder wellbeing” most prominently influenced the final evaluation of risk. Therefore, evaluations ascribed to this criterion should be considered. A comparison of single value evaluations and evaluations using the degrees of membership as assigned by the participants for the relevant criteria is presented in Table 5.

Table 5 Single Value Evaluations and Degrees of Membership of the Criterion “Stakeholder wellbeing” as Ascribed by the Participants

The evaluation of criteria using degrees of membership in the first presented evaluation (risk of patient fall) demonstrates a significantly large spectrum of possible outcomes that the participant considers when evaluating risk severity, ranging from minor to potentially critical evaluations. While in the second case (risk of employee burnout), the participant seemed to anticipate three diverse possible event outcomes, two being similarly plausible. This resulted in two distinct levels of risk with high degrees of membership, as observed in our results.

Discussion

The presented healthcare risk evaluation model has important implications for an improved risk evaluation process in healthcare organizations.

First, the evaluation model enables the evaluator to acknowledge that the consequences of an individual event can have consequences of unequal severity in different domains or areas of organizational operations. Although the risk matrix can consider the diverse consequences of an event and a holistic evaluation of risk consequences is encouraged in healthcare organizations,13 the method itself is incapable of properly acknowledging simultaneous occurrences of these impacts, since no guidance for their contribution and overall aggregation of “Severity” score is provided by the approach. This problem was recognized in an extensive review of risk evaluation practices in England that detected a lack of guidance for the use of the risk matrix approach in instances where risk has several consequences in multiple domains.41 The presented evaluation model, on the contrary is not limited by the two main criteria but can more transparently and accurately acknowledge the simultaneous impacts of risk on multiple domains of risk consequences by using additional criteria and predetermined utility functions. With its design, the evaluation model encourages the evaluator to account for all aspects of the risk consequences and provides support for determining their impact on the final evaluation. Additionally, criteria are connected with individually determined decision rules, enabling adequate precision and possible model modification in accordance with future empirical findings or organizational goals.

Second, the decision model provides more precise guidelines for criteria evaluation, enabling the evaluator to select a more suitable and consistent option when evaluating risks, thereby boosting the reliability of the evaluation process. This was evident in our results, where the participants knowingly miscategorized their decisions using the risk matrix approach to somehow integrate their knowledge into the decision-making process. The decision to emphasize the differences in likelihoods between the evaluated risks is sensible; however, it is inaccurately and untransparently handled using the risk matrix approach. Additional criteria and tailored domain values enable evaluators to integrate their knowledge more accurately, sufficiently, and transparently into the evaluation process. Even if the evaluator conducts an in-depth holistic analysis of risk to formulate a precise evaluation of the two criteria using the risk matrix, this is not evident when analyzing the risk evaluation results. Increased transparency is another clear contribution of the evaluation model, which our results clearly demonstrate. An increase in the transparency of the decision process enables an improved understanding of the evaluation results and analysis to support more precise measures for risk-mitigation strategies. As can be seen in our results, decision makers can use the evaluation model results not only to understand the rationale behind the final risk-level determination but also to recognize crucial future efforts for risk reduction. By analyzing the areas of impact, intel of the best strategies for improvement is granted. By recognizing these features of risk evaluation as useful tools, additional fronts for future improvement and research in the field of risk management in healthcare can be developed.

Third, we identified categorization uncertainty as an important barrier to adequate risk consequences evaluation in healthcare. This arises from the nature of healthcare risks, which produce not one but a spectrum of possible outcomes. In our evaluation model, this was addressed using partial degrees of membership. This functionality enables the evaluator to express diverse plausible scenarios that can occur following an event. Considering the diverse plausible outcomes of an event is a well-established approach in risk management and risk analysis methods. Using partial degrees of membership to express probability allows the evaluation model to transparently integrate findings from a preliminary in-depth analysis of risk into the evaluation process. Examples of recognized risk analysis methods that utilize this approach include Event Tree Analysis and Bowtie Analysis. They use clinical data and expert knowledge to help determine different probabilities for a range of possible outcomes of an event and have previously been used for healthcare risk analysis.31,42,43 The important contributions of these risk analysis methods are neglected or subjectively generalized in risk evaluation if the evaluator is obliged to select only one option for evaluating the risk criteria.

Previous work in this field already includes multicriteria decision-making and fuzzy logic integration to boost the quality of the risk evaluation process.29,33,44 However, some advantages of the presented model can be pointed out. The presented model uses multicriteria decision-making to enhance the sensitivity and structure of the evaluation process, while at the same time providing a fuzzy logic mechanism to address uncertainty. Another important advantage of the presented model is its wide applicability for diverse risks in healthcare evaluation, which, in comparison to existing solutions, enables it to adequately replace the use of the existing risk matrix approach.

Furthermore, our results show that risk characteristics play an important role in determining the best ways of addressing uncertainty. The use of degrees of membership enables the evaluator to implicate the examined probabilities directly into the evaluation process. This paper demonstrates the importance of appropriate uncertainty handling in this and similar decision problems, as this uncertainty cannot be adequately addressed using the more common fuzzification of linguistic expressions. Our results show that the distributions of the probability of event outcomes in healthcare can be diverse and, in some cases, highly skewed or irregular. Comparing evaluations using degrees of membership and single value evaluations also revealed two distinct approaches to handling uncertainty when obligated to select a single value between the participants. In the first case, the participant selected, in his opinion, the most probable option considering the later provided degrees of membership. However, in the case of the risk of employee burnout, the participant used a more conservative approach and seemed to evaluate risk severity according to the worst-case scenario using single values evaluation. Because of diverse distributions of outcomes probabilities and differences in style of generalization between evaluators, forced single value generalization should be avoided, making discrete probabilities a more appropriate choice, even considering that commonly exact probabilities are assumed in this approach.40 Our findings reveal a potential research front, to further highlight how uncertainties regarding the plausible outcome scenarios influence single value risk evaluation on how they should be addressed.

Results of the presented study support the usability of our approach for enhancing risk evaluation practices not only within healthcare settings but also across a wide range of industries and disciplines. The used DEX method utilizes individual rule configuration, which is highly adaptable and easy for users to understand. This enables precise expert knowledge modelling and supports potential modifications when needed. Likewise, the approach of handling uncertainties using partial degrees of membership has proven to be effective in addressing the nuanced characteristics of risks and was sufficiently adopted and understood by the participants. Insights from our study highlight the need for future research in diverse areas of risk evaluation practices, since this approach could be highly usable, particularly in similar highly complex environments where risk evaluation is inherently prone to uncertainty.

Our study has three main limitations. Although the model tackles uncertainty using degrees of membership in consequence evaluation, it does not acknowledge epistemological uncertainty, as consequence of lacking expert’s knowledge for likelihood estimation. This is an important limitation, since recognizing lacking experts’ knowledge is crucial in healthcare risk management, where accurate likelihood estimation is often limited by poor data availability. This is the result of poor safety culture and healthcare workers’ unwillingness to report adverse events for which they feel responsible. Data regarding adverse events are an important resource in risk management but are commonly under reported and lacking.6,45 Although healthcare quality managers can enhance adverse event reporting and compensate through proactive approaches in risk management,46 inadequate data can lead to subjective and unreliable likelihood estimations. Lack of support of the evaluator, both in the case of addressing uncertainties regarding likelihood and uncertainties regarding the evaluation of consequences, was also flagged by the previously cited study analyzing existing practices in England.41 The possibilities of addressing likelihood estimation-related uncertainties using fuzzification of linguistic expressions or integrating degrees of estimation confidence should be addressed in future research on this topic.

The second limitation of our study is the lack of empirical evidence to support the usability of the evaluation model in clinical practice. Future research should prioritize longitudinal study designs to evaluate the impact of the novel risk evaluation model’s actual impact on risk management practices and patient safety outcomes. Additionally long-term implementation could reveal potential barriers to successful adoption in clinical practice, since the increased complexity and time consumption of the proposed approach may hinder its usability and user acceptance, demanding adaptations of the evaluation model.

Although our results indicate greater transparency in risk evaluation, providing evidence to support its accuracy or validate the consistency of the proposed evaluation model is beyond the scope of this study. This third limitation of our study is evident in the high variance in evaluations, both between participants and across risks, highlighting the need to evaluate possible additional measures to increase objectivity. Adding additional criteria and stricter domain values description could promote objectivity, however these adaptations should be considered with caution, since they may hinder the capability of evaluators to confidently express their knowledge regarding the evaluated risk and increase time and expertise needed to produce an evaluation. This could negatively impact the usability, scalability and user acceptance of the proposed evaluation model.

Possible additional approach of addressing the limitation of interrater reliability is incorporation of multiple decision-makers’ inputs. Although interprofessional collaboration was used in the model development process, group decision-making could be more extensively utilized in model development and risk evaluation process. Existing research in this field shows that group decision-making can provide not only an improved process of evaluation model conceptualization, development and adaptations, but can also boost objectivity of the evaluation process.47,48

Besides group decision-making, study design should also be considered to address these issues in future research. Since the participants in our study originated from diverse clinical settings, poor and insignificant correlations between evaluators were anticipated. High variance between evaluators could be addressed with an improved study protocol, which should include more comprehensive training for participants and include multiple evaluators originating from same or similar clinical background, since our sample of participants was ill-suited for interrater reliability analysis. Findings from this kind of research approaches could provide more information on the evaluation model validity, consistency and applicability. Further, multicenter studies will also determine whether the selected criteria are sufficient for the use of the model in diverse healthcare organizations and settings. In the model development process, risk management practice data from a single clinic were used to identify relevant criteria.

Nevertheless, the proposed evaluation model demonstrates possibilities for future development and supports precise adaptations of the model in the future. The presented approach is not limited to healthcare-related risks evaluation but can be used in diverse fields of expertise to improve risk management practices.

Conclusion

Risk management is an important activity for quality improvement in healthcare. We present some issues with existing risk evaluation methods and a qualitative multicriteria evaluation model that addresses some of these issues. The presented study was conducted in a single clinic with a low number of participants. The scope of the presented study is therefore limited, and additional longitudinal research is needed to validate the usability and impact of the presented evaluation model in clinical practice. Future research should also consider the performance of the evaluation model in diverse healthcare organizations and settings. Adaptations and cost–benefit analyses for specific clinical environments are warranted and will provide crucial information on barriers to implementation and scalability of the approach. Even so, the proposed model improves the process of risk evaluation by providing a more transparent and precise decision-making process, while supporting the evaluator in tackling uncertainties inherent to the nature of risks in healthcare. The DEX method used for model development is highly adaptable and enables nuanced modifications of criteria structure and decision rules for the precise modelling of both expert and empirical knowledge. Consequently, the proposed approach is not limited to healthcare risk evaluation but can be applied in diverse risk evaluation environments and modified according to specific risk characteristics and evaluator needs.

Funding

The authors acknowledge financial support from the Slovenian Research Agency (research core funding No. P5-0018).

Disclosure

The authors report no professional or financial conflicts of interest in this work.

References

1. Institute of medicine (US) committee on quality of health care in America. In: Kohn LT, Corrigan JM, Donaldson MS, editors. To Err Is Human: Building a Safer Health System. Washington, DC: National Academies Press; 2000. p. 26–48.

2. Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ. 2016;353. doi:10.1136/bmj.i2139

3. Mazer BL, Nabhan C. Strengthening the medical error “meme pool”. J Gen Intern Med. 2019;34(10):2264. doi:10.1007/s11606-019-05156-7

4. Avery AJ, Sheehan C, Bell B, et al. Incidence, nature and causes of avoidable significant harm in primary care in England: retrospective case note review. BMJ Qual Saf. 2021;30(12):961–976. doi:10.1136/bmjqs-2020-011405

5. Newman-Toker DE, Nassery N, Schaffer AC, et al. Burden of serious harms from diagnostic error in the USA. BMJ Qual Saf. 2024;33(2):109–120. doi:10.1136/bmjqs-2021-014130

6. Rodziewicz TL, Houseman B, Vaqar S, Hipskind JE. Medical error reduction and prevention. In: StatPearls [Internet]. Treasure Island, FL: StatPearls Publishing; 2024.

7. Tello JE, Barbazza E, Waddell K. Review of 128 quality of care mechanisms: a framework and mapping for health system stewards. Health Policy. 2020;124(1):12–24. doi:10.1016/j.healthpol.2019.11.006

8. Alhawajreh MJ, Paterson AS, Jackson WJ. Impact of hospital accreditation on quality improvement in healthcare: a systematic review. PLoS One. 2023;18(12):e0294180. doi:10.1371/journal.pone.0294180

9. American Accreditation Commission International. Accreditation Standards for Healthcare Organizations; 2019.

10. Björnsdóttir SH, Jensson P, de Boer RJ, Thorsteinsson SE. The importance of risk management: what is missing in ISO standards? Risk Anal. 2022;42(4):659–691. doi:10.1111/risa.13803

11. International Organization for Standardization. Risk management: guidelines. ISO. 2018;31000:2018.

12. Ferdosi M, Rezayatmand R, Taleghani YM. Risk management in executive levels of healthcare organizations: insights from a scoping review (2018). Risk Manag Healthc Policy. 2020;13:215–243. doi:10.2147/RMHP.S231712

13. Pascarella G, Rossi M, Montella E, et al. Risk analysis in healthcare organizations: methodological framework and critical variables. Risk Manag Healthc Policy. 2021;14:2897–2911. doi:10.2147/RMHP.S309098

14. Duijm NJ. Recommendations on the use and design of risk matrices. Saf Sci. 2015;76:21–31. doi:10.1016/j.ssci.2015.02.014

15. Lane K, Hrudey SE. A critical review of risk matrices used in water safety planning: improving risk matrix construction. J Water Health. 2023;21(12):1795–1811. doi:10.2166/wh.2023.129

16. Garvey PR, Lansdowne ZF. Risk matrix: an approach for identifying, assessing, and ranking program risks. Air Force J Logist. 1998;22(1):18–21.

17. Qazi A, Shamayleh A, El-Sayegh S, Formaneck S. Prioritizing risks in sustainable construction projects using a risk matrix-based Monte Carlo simulation approach. Sustain Cities Soc. 2021;65:102576. doi:10.1016/j.scs.2020.102576

18. Qin M, Liao K, He G, He T, Leng J, Zhang S. Quantitative risk assessment of static equipment in petroleum and natural gas processing station based on corrosion-thinning failure degree. Process Saf Environ Prot. 2023;172:144–156. doi:10.1016/j.psep.2023.01.045

19. Allendorf V, Denzin N, Conraths FJ, et al. Does having a cat in your house increase your risk of catching COVID-19? One Health. 2022;14:100381. doi:10.1016/j.onehlt.2022.100381

20. Gao Z, Liu Q, Yang L, Zhu X. Identification of high-risk factors for prehospital delay for patients with stroke using the risk matrix methods. Front Public Health. 2022;10:858926. doi:10.3389/fpubh.2022.858926

21. Lemmens SMP, Lopes van Balen VA, Röselaers YCM, Scheepers HCJ, Spaanderman MEA. The risk matrix approach: a helpful tool weighing probability and impact when deciding on preventive and diagnostic interventions. BMC Health Serv Res. 2022;22(1):218. doi:10.1186/s12913-022-07484-7

22. Williams K, Cherrie JW, Dobbie J, Agius RM. The development of a Covid-19 control measures risk matrix for occupational hygiene protective measures. Ann Work Expo Heal. 2022;66(2):269–275. doi:10.1093/annweh/wxab050

23. La C Jr. What’s wrong with risk matrices? Risk Anal. 2008;28(2):497–512. doi:10.1111/j.1539-6924.2008.01030.x

24. Thomas P, Bratvold RB, Bickel JE. The risk of using risk matrices. Paper presented at: SPE Annual Technical Conference and Exhibition; September, 2013; New Orleans, Louisiana.

25. Ball DJ, watt J. Further thoughts on the utility of risk matrices. Risk Anal. 2013;33(11):2068–2078. doi:10.1111/risa.12057

26. Vatanpour S, Hrudey SE, Dinu I. Can public health risk assessment using risk matrices be misleading? Int J Environ Res Public Health. 2015;12(8):9575–9588. doi:10.3390/ijerph120809575

27. Elmontsri M. Review of the strengths and weaknesses of risk matrices. J Risk Anal Crisis Response. 2014;4(1):49–57. doi:10.2991/jrarc.2014.4.1.6

28. Meziane S, Taous M, Bourkhiss B. Mitigating the risk of COVID-19 infection in the operating room: using healthcare failure mode and effect analysis. Perioper Care Oper Room Manag. 2023;31:100319. doi:10.1016/j.pcorm.2023.100319

29. Sabripoor A, Ghousi R, Najafi M, Barzinpour F, Makuei A. Risk assessment of organ transplant operation: a fuzzy hybrid MCDM approach based on fuzzy FMEA. PLoS One. 2024;19(5):e0299655. doi:10.1371/journal.pone.0299655

30. Chakraborty S, Raut RD, Rofin TM, Chakraborty S. A comprehensive and systematic review of multi-criteria decision-making methods and applications in healthcare. Healthcare Anal. 2023;4:100232. doi:10.1016/j.health.2023.100232

31. Culwick MD, Endlich Y, Prineas SN. The Bowtie diagram: a simple tool for analysis and planning in anesthesia. Curr Opin Anaesthesiol. 2020;33(6):808–814. doi:10.1097/ACO.0000000000000926

32. Sahoo SK, Goswami SS. A comprehensive review of multiple criteria decision-making (MCDM) methods: advancements, applications, and future directions. Decision Making Advances. 2023;1(1):25–48. doi:10.31181/dma1120237

33. Ghanem MAAN, Zaifoglu H. A geospatial analysis of flood risk zones in Cyprus: insights from statistical and multi-criteria decision analysis methods. Environ Sci Pollut Res Int. 2024;31(22):32875–32900. doi:10.1007/s11356-024-33391-x

34. Bohanec M. DEX (decision expert): a qualitative hierarchical multi-criteria method. In: Kulkarni AJ, editor. Multiple Criteria Decision Making: Techniques, Analysis and Applications. Singapore: Springer Nature Singapore; 2022:39–78.

35. Boshkoska BM, Miljković D, Valmarska A, et al. Decision support for medication change of Parkinson’s disease patients. Comput Methods Programs Biomed. 2020;196:105552. doi:10.1016/j.cmpb.2020.105552

36. Jozef Stefan Institute. DEXi: A program for multi-attribute decision making. 2024. Available at: https://dex.ijs.si/dexisuite/dexisuite.html. Accessed August 2, 2024.

37. Zadeh LA. Fuzzy sets. Inf Control. 1965;8(3):338–353. doi:10.1016/S0019-9958(65)90241-X

38. Celikyilmaz A, Türksen IB. Modeling Uncertainty With Fuzzy Logic. Vol. 240. Berlin Heidelberg: Springer; 2009.

39. Kruse R, Mostaghim S, Borgelt C, Braune C, Steinbrecher M. Computational Intelligence. Springer International Publishing; 2022.

40. Hawer S, Schönmann A, Reinhart G. Guideline for the classification and modelling of uncertainty and fuzziness. Procedia CIRP. 2018;67:52–57. doi:10.1016/j.procir.2017.12.175

41. Kaya GK, Ward J, Clarkson J. A review of risk matrices used in acute hospitals in England. Risk Anal. 2019;39(5):1060–1070. doi:10.1111/risa.13221

42. Liebmann G, Schuster G. Fuzzy system based event tree analysis support of the smart security system of health care centers and hospitals. Interdiscip Des Complex Syst. 2022;20(3):239–249. doi:10.7906/indecs.20.3.3

43. McLeod R, Russell W, Stewart M, Prentice M, Bowie P. Preliminary case report study of training and support needed to conduct bowtie analysis in healthcare. BMJ Open Qual. 2021;10(2):e001240. doi:10.1136/bmjoq-2020-001240

44. Pham LT, Hoang LV. A navigational risk evaluation of ferry transport: continuous risk management matrix based on fuzzy best-worst method. PLoS One. 2024;19(9):e0309667. doi:10.1371/journal.pone.0309667

45. Zhang X, Ma S, Sun X, et al. Composition and risk assessment of perioperative patient safety incidents reported by anesthesiologists from 2009 to 2019: a single‐center retrospective cohort study. BMC Anesthesiol. 2021;21(1):8. doi:10.1186/s12871-020-01226-0

46. Yuan B, Song S, Tang X, Ma Z. Application of multidimensional quality management tools in the management of medical adverse events. Risk Manag Healthc Policy. 2024;17:91–99. doi:10.2147/RMHP.S436083

47. Ghasemi F, Rahimi J. Failure mode and effect analysis of personal fall arrest system under the intuitionistic fuzzy environment. Heliyon. 2023;9(6):e16606. doi:10.1016/j.heliyon.2023.e16606

48. Fu S, Xiao YZ, Zhou HJ. Interval-valued intuitionistic fuzzy multi-attribute group decision-making method considering risk preference of decision-makers and its application. Sci Rep. 2022;12(1):11597. doi:10.1038/s41598-022-15815-1

Creative Commons License © 2025 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, 3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.