Development and Validation of an Indicator System for Evaluating Clinical Nursing Process Quality Using Mobile Nursing Information Systems

Introduction

The quality of nursing processes, which reflects the performance of each stage in nursing activities, directly impacts the overall quality of nursing care.¹ The Donabedian three-dimensional theory² of quality posits that process quality directly influences end quality. Monitoring indicators for nursing process quality enables proactive oversight, aiding nursing managers in identifying issues promptly and implementing appropriate interventions. This approach ensures the standardization and normalization of nursing practices.³ Traditionally, nursing process quality has been assessed using on-site evaluation methods, such as direct observation, chart reviews, and audits^4,5 While these methods provide valuable insights, they are often time-intensive, resource-intensive, and limited to spot checks. These limitations impede the collection of large-scale data and the implementation of multi-node quality control, making comprehensive management of nursing process quality challenging.

Recent advancements in technology have introduced alternative approaches, such as internet-based information technologies and mobile nursing information systems, which offer more efficient and scalable solutions for nursing quality management.⁶ Internet-based information technologies refer to digital tools and platforms that leverage the internet to collect, analyze, and disseminate data. In nursing, these technologies enable real-time monitoring and evaluation of care processes, enhancing transparency and objectivity. Mobile nursing information systems, a subset of these technologies, are designed to capture operational records of nursing activities at each stage, facilitating data extraction and analysis in accordance with quality control standards. The Opinions of the General Office of China State Council on Promoting the High-Quality Development of Public Hospitals (2021)⁷ emphasized the importance of leveraging internet-based information technologies to enhance objectivity and transparency in nursing quality evaluation and management. The Implementation Guidelines for Evaluation Standards of Tertiary-Grade Hospitals (2020 Edition)⁸ emphasizes the use of information technology to enhance daily supervision, establish objective evaluation indicators, and incorporate key, reliable quantitative measures as assessment criteria. At present, some hospitals at home and abroad have begun research on the use of hospital information system data to carry out quality evaluation. Han et al utilized the hospital information system to identify a set of key indicators of quality. They then employed the Delphi method to construct a nursing quality evaluation system, identifying a data set of nursing quality indicators with the ward nursing unit as the basic unit. This data set contained 33 tertiary indicator items. The application of this evaluation index system has been shown to improve patient satisfaction, nurse turnover rate, medication administration accuracy rate, and the number of cases of adverse events. However, the study focused on fewer process indicators, which could not fully reflect the process quality management of clinical nursing activities, and did not adequately utilise the information data.^9–11 Fang et al analysed the nature of data from multiple information systems in hospitals, sorted out the key operational steps of items related to the nursing process, established quality control points, and constructed informatization process practices for clinical nursing activities such as the whole process of drug use, the whole process of blood transfusion, and the whole process of safe perioperative patient transfer. The construction of process indicators is closer to clinical practice from the perspective of quality control of key links, but the study only mentions some clinical nursing activities and lacks objective evaluation of its application effect.¹² The People’s Hospital of Peking University adopted an in-hospital information system, PDA barcode technology and mobile nursing trolleys to achieve closed-loop management of several clinical care processes. The selected process indicators encompassed medication execution scanning and confirmation, risk assessment, the completion rate of 24-hour health education on admission and the documentation of nursing activities. However, the system also exhibited deficiencies that hindered its ability to adequately reflect the quality management of the clinical care process.¹³ Jiang et al used the mobile nursing quality control management information technology platform for nursing quality control, and the results showed that the application of the mobile nursing quality control management information technology platform could shorten the time cost, reduce the inspection time-consuming, and improve the work efficiency.¹⁴

These technologies improve efficiency and timeliness, enabling more comprehensive and scientific nursing quality management.^15,16 These systems enable data extraction and analysis in accordance with quality control standards, facilitating the management of nursing quality throughout the process.¹⁷

The advent of automation and intelligence in information technology, coupled with the efficiency and timeliness of information data transmission, has paved the way for the realization of timely, transparent, and equitable quality evaluation and monitoring. This development has led to a more scientific, comprehensive, and complete quality management of the nursing process, thereby serving as an effective method for continuous quality enhancement. However, it should be noted that current studies in this domain are beset with the following challenges:

(1) While the majority of studies emphasize the use of information data, the process indicators do not generally achieve direct automatic capture of data, and there is an absence of specific calculation of the indicators.(2) With regard to the construction and selection of process indicators, these are mostly based on experience or hospital level assessment requirements, and are less supported by theoretical frameworks, which cannot reflect the comprehensive and complete process quality management from the perspective of information data utilization.¹⁸

The Clinical Care Classification (CCC) system,^19,20 a standardized nursing terminology recognized by the American Nurses Association (ANA), categorizes nursing activities into four types: assessment, execution, guidance, and management. The CCC system provides a structured framework for documenting and evaluating nursing care, making it a valuable tool for quality management. We employ the CCC system as a foundational framework. Primarily, the CCC system has the capacity to encompass the entire nursing process in its entirety, thereby providing comprehensive coverage of all aspects of nursing quality. Each nursing activity is assigned a unique code, thus facilitating the quantification of statistics and the analysis of the execution of the nursing process. To illustrate this point, the completion of assessment categories can serve as an indicator of the comprehensiveness of nursing assessment, while the timeliness of execution categories can serve as a measure of the efficacy of nursing measures. Secondly, the electronic compatibility of the CCC system with modern nursing information systems enables real-time data analysis and quality monitoring. In addition, the CCC system complies with the requirements of the International Organization for Standardization (ISO) for nursing terminology systems, thus supporting cross-institutional data comparability. The CCC system facilitates a shift in the quality evaluation of the nursing process from “experience-driven” to “data-driven”, thereby providing a scientific basis for enhancing the safety and efficiency of nursing care. However, its application in nursing process evaluation remains underexplored, particularly in the context of information system-based approaches.

This study aims to develop an indicator system for assessing the quality of clinical nursing processes within the CCC system framework. The proposed system is designed to be objective, measurable, and efficient, leveraging advanced mobile nursing information systems to focus on the quality management of clinical nursing processes. Specifically, the study seeks to identify key indicators for evaluating nursing process quality within the CCC system framework and to validate the proposed system through pilot testing in a clinical setting. The findings of this study are expected to provide a theoretical foundation for the objective and comprehensive evaluation of clinical nursing processes, addressing existing gaps in nursing quality management.

Data and Methods

Construction of the Indicator System for Quality Evaluation of Clinical Nursing Processes Based on a Mobile Nursing Information System

Establishment of the Study Team

A multidisciplinary team was established, consisting of six members: three associate chief nurses, two supervising nurses, and one registered nurse. The team included a deputy director of the Nursing Department, a quality control specialist from the Nursing Department, two clinical nursing experts, a nursing information management professional, and a nurse with a master’s degree. All of the study team work for the same organisation.

Literature Review

A systematic literature search was performed across multiple databases, including China National Knowledge Infrastructure (CNKI), Wanfang Med Online, CQVIP, PubMed, and Web of Science. The search covered the period from the inception of each database to March 2024. The search terms included “quality of nursing process”, “quality evaluation”, “mobile nursing”, and “information”, in Chinese and English. The scope of clinical nursing activities in hospital wards was defined based on the four nursing activity types outlined in the CCC System and national policy documents, including the Implementation Guidelines for Evaluation Standards of Tertiary-Grade Hospitals (2020 Edition) and the Regulations on Quality Management in Medical Institutions.

Drafting the Initial Expert Correspondence Consultation Questionnaire

The mobile nursing information system is an existing technology-driven platform developed by a software developer. It is designed to support nursing workflows and has been customized by the hospital to tailor various functional modules to meet specific needs. Targeted at the nursing staff, the application is primarily used for medical order execution and nursing documentation. Additionally, it allows for the extraction of nursing work data from the backend. Hospital information system personnel responsible for the nursing information system were consulted to verify the accessibility of data. Data extraction was subsequently performed. Based on the extracted data, all closed-loop path nodes of clinical nursing activities in the information system were identified, organized and summarized. The study team, in collaboration with an expert panel, reviewed, revised, and integrated by these nodes through discussions. The information nodes of nursing activities were transformed into specific evaluation indicators, resulting in an initial draft of quality evaluation indicators for clinical nursing processes. An expert consultation questionnaire was developed, using a 5-point Likert scale to assess the importance of each indicator, where 1 represented “not important” and 5 indicated “very important”.

Enrollment of Consultation Experts

Experts were recruited according to the inclusion and exclusion criteria outlined by the Delphi method.²¹ Experts with recognized authority and extensive experience in clinical nursing, nursing information systems, or nursing management were selected. The initial target was to enroll 20 experts.

Inclusion criteria were as follows: (1) Experts working in the fields of clinical nursing, nursing management, nursing quality management, nursing information, or information technology; (2) Possess a bachelor’s degree or higher; (3) Hold an intermediate or higher professional title; (4) A minimum of five years of relevant work experience; and (5) Willingness to participate in the study.

Expert Consultation via Correspondence

In the present study, we used a two-round Delphi method. In the first round, the questionnaires were distributed to all consultation experts through Email or WeChat, with a request to complete and return them within seven days. The questionnaire included the initial list of indicators, and the experts were asked to rate the indicators according to the importance on a Likert scale from 1 (not important) to 5 (very important). A higher score indicated a higher level of importance. During the consultation process, questions raised by the experts were addressed via telephone communication. The second round of expert correspondence questionnaire was formed by adding and deleting entries based on the screening criteria for each indicator entry (importance score ≥3.5, coefficient of variation ≤0.25) and combining the relevant opinions of the experts. Following two rounds of expert correspondence consultation, a gradual convergence of views among the experts was observed. The development of an indicator system for the quality evaluation of clinical nursing processes, utilising a mobile nursing information system, was finalized. The system comprises four primary indicators and 44 secondary indicators.

Statistical Methods

Data collection and analysis were conducted using Excel and SPSS Statistics 29.0. The importance scores of indicators are expressed as the mean ± standard deviation. The level of expert participation was assessed through the effective return rate of the questionnaires, while the authority of experts was assessed using the expert authority coefficient (Cr), calculated as Cr=(Ca+Cs)/2 Cr=(Ca+Cs)/2, where Ca represents the coefficient of judgment criteria and Cs denotes the coefficient of familiarity. A Cr value of ≥ 0.70 was considered indicative of acceptable reliability, ensuring the robustness and credibility of the expert consultation outcomes, forming a reliable foundation for the development of the indicator system.

The degree of consensus among experts was measured using Kendall’s coefficient of concordance (Kendall’s W), with values ranging from 0 to 1, where a larger W value indicated a higher level of consensus. A P-value of < 0.05 was considered statistically significant. This statistical measure helps to verify that the indicators selected are widely endorsed by the expert panel, thus reinforcing the validity of the indicator system.

The indicator weights were determined using a precedence ordering chart.²² Importance scores assigned by experts were subjected to pairwise comparisons, and a weight calculation table was generated, where a smaller score was assigned 0 points, an equal score 0.5 points, and a larger score 1 point. The relative score for each indicator was calculated as the sum of its row values in the weight calculation table. The weight of each indicator was then derived by dividing its relative score by the total relative scores of all indicators, ensuring that each indicator’s significance is proportionally represented, thereby enhancing the precision and applicability of the indicator system for evaluating clinical nursing process quality.

Extraction and Application of Quality Evaluation Indicators for Clinical Nursing Processes Based on a Mobile Nursing Information System

Extraction of Indicator Data

Indicator data were extracted from the backend of the hospital’s information system for three clinical departments covering the period of May and June 2024. Each indicator was defined, and relevant calculation formulas were developed based on the Regulations on Quality Management in Medical Institutions, Indicators of Medical Quality Control in Nursing, and other relevant guidelines. Data extraction for each indicator was conducted by information technology professionals. Each calculation formula corresponded to a specific dataset. For instance, the calculation formula for the “Execution rate of medical orders for blood/urine/stool/sputum specimen collection” is: , Specimen collection involved the following operational steps: scanning the barcode on the specimen collection tube using a Personal Digital Assistant (PDA), scanning the patient’s wristband Quick Response (QR) code, and selecting the “collection completion” button. Information technology professionals developed algorithms to extract two data points from the information system: the total number of medical orders for specimen collection and the total number of completed specimen collections recorded through the “collection completion” action on the PDA. These data points constituted the dataset for calculating the “execution rate of medical orders for blood/urine/stool/sputum specimen collection”.

All indicator datasets were compiled and organized into an Excel file, which was double-checked by both authors and information technology professionals. Any doubtful data were traced back to the source to ensure accuracy and authenticity. Once the data was verified, authors calculated the execution rate for each indicator using the established calculation formula.

Evaluation Method

Indicators ranking in the top half based on their integrative weights within each hierarchical level were identified through discussions by the study team. Data for these selected indicators were extracted from the system’s backend. The execution status of these indicators across the three clinical departments was statistically analyzed, and differences in their execution status among the departments were compared. The practicality and feasibility of the indicator system for assessing the quality of clinical nursing processes, based on a mobile nursing information system, were validated using objective data.

Results

Basic Information of Experts Consulted via Correspondence

A total of 20 experts from clinical nursing, nursing management, nursing information, and related fields participated in this study. The age of the experts ranged from 36 to 50 years, with a mean age of 43.85 ± 4.80 years. Their work experience varied from 11 to 33 years, with an average age of 21.50 ± 7.31 years. Among the participants, 14 held bachelor’s degrees and 6 held master’s degrees. The professional titles included 16 associate chief nurses and 4 supervising nurses. In terms of roles, 7 were department head nurses, 11 were head nurses under the supervision of department heads, and 2 were staff members of the nursing department.

Coefficients of Expert Participation, Expert Authority, Consensus, and Convergence of Opinions

For both rounds of expert correspondence consultation, 20 questionnaires were distributed and 20 were returned, achieving a valid return rate of 100% in each round, reflecting a high level of expert participation. The coefficients of judgment criteria Ca, Cs, and Cr were 0.92 and 0.93, 0.90 and 0.84, and 0.89 and 0.90 for rounds one and two, respectively, indicating reliable consultation results. The values of Kendall’s W were 0.210 for the first round and 0.129 for the second round (both P < 0.05), signifying a high degree of consensus among experts and confirming the reliability of the consultation process. The results from both rounds of consultation demonstrated that the mean importance score for each indicator ranged from 3.85 to 5.00, with coefficients of variation between 0.00 and 0.24 (Table 1).

Table 1 Results of Expert Consultation via Correspondence on the Indicator System for Evaluating Clinical Nursing Process Quality Using a Mobile Nursing Information System

Evaluation Indicator System and Weight Results

The finalized indicator system for evaluating the quality of clinical nursing processes, based on a mobile nursing information system, consists of 4 primary indicators and 44 secondary indicators. Table 2 provides details on each indicator, including the mean importance, mean and standard deviation, coefficient of variation, and integrative weight. The consistency index (CI) and consistency ratio (CR) for the primary indicators were calculated as 0.063 and 0.039, respectively. The CR values for all secondary indicators were below 0.1. These consistency test results indicate that the indicator settings are reasonable and conform to the established requirements.

Table 2 Execution Status and Indicator Values for Each Department

Extraction of Indicator Data and Results of Statistical Analysis

The assessment layer indicators demonstrated the highest execution rate, consistently exceeding 98%. In contrast, the indicator of the guidance layer, specifically the “effective reading rate of content on the educational platform”, exhibited the lowest execution rate. Department 1 had significantly lower execution rates for procedures such as subcutaneous injection, intradermal injection, intramuscular injection, intravenous injection, nebulization inhalation, oral medication, and intravenous infusion compared to the other two departments (P < 0.05). Department 2 showed significantly lower rates for the timely submission of blood, urine, stool, and sputum specimens for testing compared to the other departments (P < 0.05) (Table 2).

Discussion

The literature analysis indicates that leveraging mobile nursing information systems for quality control can significantly boost clinical nursing quality and decrease nursing error rates.¹⁴ However, existing research has largely overlooked nursing process quality indicators, often focusing on single institutions or specific scenarios. This has led to a lack of standardized indicators and limited comparability across different institutions.¹³ To address these gaps, this study develops a nursing process quality indicator system based on the hospital’s existing mobile nursing information system and guided by the CCC System. This framework ensures comprehensive coverage of all nursing activities and enhances the cross-scenario applicability of the indicators.

Strengths and Implications of the Quality Indicator System for Clinical Nursing Processes

This study developed a scientifically sound and feasible indicator system for assessing the quality of clinical nursing processes. The foundation for this system was built on a comprehensive literature review and the CCC System, which provided a theoretical framework for indicator selection. The indicator system for assessing the quality of clinical nursing processes developed comprised four primary indicators and 44 secondary indicators. This system was developed through expert consultation using the Delphi method and discussions among the study team, ensuring a scientific and rigorous development process.

The results of this study confirm the expertise of the consulted experts, professionals from clinical nursing, nursing informatics, and nursing management, each with over 10 years of relevant experience. The expert participation rate was 100% in both rounds of consultation, as reflected by the effective return. The expert authority coefficients were calculated as 0.89 and 0.90 for the two rounds, respectively, with 12 experts providing suggestions for indicator modification. These findings highlight a high level of expert engagement and authority in the consultation process. Kendall’s W for the two rounds were 0.210 and 0.169, respectively, demonstrating a relatively high level of consensus among experts and strong reliability of the consultation results.

The quality evaluation indicators of the nursing process should possess several major characteristics, including objectivity, measurability, and data accessibility.²³ Previous studies in nursing process evaluation often lacked a robust theoretical framework, focused on a limited range of indicators, and exhibited potential bias, thereby failing to objectively reflect the quality management of clinical nursing processes. In contrast, the indicator system developed in this study is constructed within the theoretical framework of the CCC System, encompassing four types of clinical nursing activities: assessment, execution, guidance, and referral. This approach ensures a comprehensive and objective evaluation of nursing process quality.²⁴

The calculation methods for the indicators were clearly defined based on their underlying conceptual definitions and aligned with the mobile nursing information system’s data model. This alignment facilitated reliable data collection, ensuring accurate measurement of indicators. This approach supports the ease of indicator measurement, data accessibility, and operational feasibility.

Rationale for Weight Assignments in a Nursing Quality Indicator System

In this study, the weight of each indicator was determined using a precedence ordering chart, which provided a framework for prioritizing nursing activities during nursing quality evaluation. This weight assignment method ensures that the most critical aspects of nursing care are given priority in the evaluation. The results of the weight analysis highlighted that the primary indicator of assessment/monitoring/evaluation/observation demonstrated the highest weight value (0.438), with a mean importance score of 5.00. Experts reached a unanimous agreement on its critical importance. Nursing assessment is foundational to the implementation of effective nursing interventions and serves as a key indicator of nursing quality. Accurate and comprehensive nursing assessments enable healthcare providers to identify the care needs of patients and evaluate their status systematically during routine nursing activities.²⁵

The indicator execution/nursing/provision/assistance ranked second among the primary indicators, with a weight value of 0.313. This indicator primarily evaluates the standardization and timeliness of nursing procedures. Among the secondary indicators under this category, execution rates of blood transfusion-related procedures—such as blood matching execution rate, blood transfusion execution rate, blood transfusion audit execution rate, blood transfusion rounds execution rate, and execution rate of the complete blood transfusion procedure—carried the highest weight. Closed-loop management of blood transfusion is recognized as a key component of nursing quality control. Clearly defined nursing quality evaluation indicators can significantly reduce the occurrence of adverse events related to blood transfusion.²⁶

In this study, execution rates at each stage of blood transfusion were extracted from the mobile nursing information system, allowing for the identification of nursing deficiencies at specific stages. As these analyses were based on authentic and objective data, the conclusions drawn enable targeted interventions to enhance the quality control of nursing processes.

Evaluation of Nursing Process Quality Using a Mobile Nursing Information System

This study aimed to assess the quality of nursing processes across departments by comparing the execution status of various indicators. Identifying such issues enables department managers to implement targeted strategies for quality management.

The analysis of indicator execution rates indicated that the assessment/monitoring/evaluation/observation category demonstrated consistently high performance, with an overall execution rate exceeding 98.0%. In contrast, the teaching/training/guidance/supervision category revealed lower execution rates, with the patient reading rate of content on the educational platform being notably low at 66.22%. Statistically significant differences in patient reading rates were observed among the three departments. Two primary factors contributed to this outcome: inadequate patient education provided by nurses and challenges posed by patient demographics, such as age and health conditions. Many older adult patients either lack access to smartphones or are unfamiliar with using digital platforms, resulting in lower reading rates.

Patient health education and follow-up are essential indicators of overall nursing quality. It is recommended that nursing managers reinforce the standardized use of educational platforms by nurses and conduct evaluations of their educational performance that align with clinical realities.

The statistical analysis of indicator execution status across departments indicated that Department 1 exhibited significantly lower execution rates for subcutaneous, intradermal, intramuscular, and intravenous injections, nebulization inhalation, oral medication, and intravenous infusion compared to the other two departments (P < 0.05). Department 2 demonstrated significantly lower rates of timely submission of blood, urine, stool, and sputum specimens (P < 0.05). These findings highlight the need for nursing managers to strengthen adherence to operational standards and implement targeted improvement measures to enhance the quality of nursing care.

Data-Driven Management of Clinical Nursing Processes

Data plays a pivotal role in the scientific enhancement of nursing quality.²⁷ In this study, data from key stages and operational procedures of nursing activities were extracted through a mobile nursing information system and analyzed. This system ensures data traceability and enables real-time monitoring of the execution and completion status of each indicator, facilitating the scientific assessment of clinical nursing quality.

The integration of informatization into nursing management underscores its importance in achieving refined and data-driven practices. The use of objective data provides a precise mechanism for identifying problems and implementing targeted improvement measures. By presenting findings through data and validating improvement outcomes, this process enables quality control to function as a dynamic, continuous, and self-improving system.

Through the extraction and analysis of indicator data across departments, several key outcomes were achieved. First, the feasibility of indicator data collection was confirmed. Second, data were found to effectively reflect the quality of nursing processes in each department. However, several challenges associated with data-based nursing quality evaluation were identified.

One notable issue is that the quality of indicator data depends heavily on the accuracy of medical orders and the standardization of nursing quality control requirements. When discrepancies in data arise, nursing managers must investigate potential non-quality-related factors that could affect the accuracy of the evaluation. Comprehensive nursing quality evaluation should integrate data from the information system with the actual clinical context.

It is also necessary to enhance the standardization of the use of mobile nursing information systems by nurses to improve the authenticity of quality indicator data. Furthermore, the execution and completion status of indicators is closely linked to the stability of the information system. This underscores the importance of ensuring system reliability, promoting adherence to scanning operation standards among nurses, and maintaining data integrity. These measures are essential for achieving accurate and reliable evaluations of nursing quality.

Limitations

This study has several limitations that should be acknowledged. Firstly, the Delphi process may introduce selection bias, as the panel of experts may not fully represent the diversity of nursing practices across different departments or regions. Future studies should aim for a more inclusive and representative panel to address this issue. Secondly, the study’s findings are based on a specific context, which may limit their applicability to other settings, such as rural hospitals or specialized care units. Further testing in diverse environments is necessary to assess the system’s broader utility. Thirdly, the current system relies on manual data capture and calculations, which can lead to errors and inefficiencies. Enhancing automation should be a priority in future iterations. Additionally, the modest sample size may limit the generalizability of the findings. Future research should incorporate a larger and more diverse expert panel to provide broader insights. While the system shows promise, its impact on actual patient outcomes and nursing quality has yet to be fully validated. Longitudinal studies are needed to assess its long-term effectiveness. Lastly, responses should be captured anonymously to encourage full and candid disclosures in future studies.

Conclusion

This study developed a scientifically grounded evaluation system for nursing process quality, leveraging a mobile nursing information system. In clinical practice, nursing managers can selectively evaluate the quality of the nursing process based on this indicator system in conjunction with their hospital’s mobile nursing system in order to evaluate the quality of the nursing process more efficiently. The findings could also influence policy by encouraging regulatory bodies to adopt standardized nursing quality evaluation frameworks and promoting the use of mobile information systems in nursing practice guidelines.