The Mayo Leadership Impact Index Adapted for Matrix Leadership Structures: Initial Validity Evidence

Introduction

Physician well-being influences quality, safety, satisfactoriness and cost of patient care,^1–4 workforce retention,^5,6 risk of malpractice lawsuits,⁷ and healthcare organization performance.^8,9 Support from an immediate supervisor is one of the biggest drivers of physicians’ satisfaction within healthcare organizations.^10,11 One study found that every unit increase in ratings of one’s leader was associated with a 9% increase in physician satisfaction and a 3% decrease in burnout.¹² Interdisciplinary and longitudinal studies replicate these findings.^13,14 In a multi-site study, physicians who rated their supervisor’s performance within the topmost tertile reported 48% lower risk of burnout, 66% lower intent to leave their organization within 2 years, and 5.8 times greater odds of high professional fulfillment.¹⁵ However, physician training typically includes little to no formal leadership development.¹⁶ Recent initiatives are incorporating leadership development into residency/fellowship training and specialists’ continuing medical education.^17–22 Such initiatives necessitate the accurate assessment and periodic tracking of targeted and beneficial leadership behaviors.^23–25

The Mayo Leadership Impact Index© (MLII), formerly known as the Mayo Clinic Participatory Management Leadership Index, is a self-report scale that assesses healthcare workers’ “direct report” supervisors across dimensions of supportive behavior such as: inclusion, keeping people informed, empowering team members, nurturing professional development, soliciting input, and providing feedback and recognition.^12,26 One of the most widely utilized measures of well-being centered leadership in healthcare organizations, the MLII was first developed and validated at Mayo Clinic, where each physician is led by a single “direct report” supervisor.²⁷ “Direct report” leadership structures exist in healthcare organizations whose administrative hierarchy assigns an immediate “frontline” supervisor to each physician. In contrast, organizations with “matrixed” leadership reporting structures have flexible hierarchies that link each physician to leaders at multiple levels such that a physician can flexibly obtain support, supervision, or mentorship from any of the potential alternative sources, based on the specific need or context. Most studies that link leadership support ratings with burnout and professional fulfillment were conducted in “direct report” settings,^13–15,25 except for graduate medical education studies where residents/fellows rate overall “program leadership” rather than a single leader.^28,29 The generalizability of these study findings to more flexible multiform leadership structures is unknown.¹⁵

Escalating consolidation of practice groups and hospitals within healthcare delivery systems in the private and academic settings has created organizational leadership structures with heterogeneous degrees of vertical and horizontal integration.^30–33 Many physicians work in loosely integrated settings with flexible, matrixed reporting structures that enable multiple and optional sources of leadership support.^34,35 This calls for an adaptation of the MLII for use in flexible, matrixed leadership structures, which then necessitates an investigation of the adapted measure’s construct validity and reliability in these settings.³⁶ The present study applied classical test theory (CTT) and item response theory (IRT) to validate an adaptation of the MLII. This is the first adaptation of the MLII for the empirical assessment of leadership performance within organizations with matrixed leadership reporting structures.

Methods and Materials

Study Design

The study was nested within a cross-sectional anonymized “quality improvement” survey. The Baylor Scott & White Research Institute Institutional Review Board waived written informed consent requirements and approved the study (# 023–171). The study adhered to STROBE³⁷ and CHERRIES³⁸ guidelines.

Participants

The study included credentialed physicians from various specialties (see Table 1) providing care to in- or out-patients across three enterprise hospitals plus associated ambulatory clinics who responded to an annual “Physician Well-being Survey”. Excluded were physicians with less than one year of organizational tenure, those with no patient-care encounters in the preceding year, and residency/fellowship trainees.

Table 1 Social Demographics and Clinical Work Characteristics of the Study Sample

Data Collection

A hyperlink to the online questionnaire was e-mailed to eligible physicians between January 24^th 2023 and February 10^th 2023. Data were managed via the Research Electronic Data Capture (REDCap™)³⁹ platform. REDCap has a proven track record as a secure and reliable web-based application for building and managing online surveys and databases specifically for research studies.

Survey Measures

Contextual Variables

Physicians’ demographics (eg, age, gender, and race/ethnicity), service location (city), department/unit, clinical experience (years in practice), and annual caseload were surveyed.

Psychosocial Variables

Standardized measures included:

Mayo Clinic Leadership Impact Index (adapted): Factor analytic studies of the original 12-item MLII version found that three items contributed minimally to the construct, and the scale was shortened to 9 items.^13,14 The Mayo Leadership Impact Index (MLII) remains proprietary to the Mayo Clinic organization and was adapted with permission. Three co-authors collaboratively proposed changes to this pre-existing 9-item revised MLII. Proposed changes were piloted among a 10-member advisory panel of physicians whose input was incorporated. Ten candidate items (adaptations of the original 9 plus a newly crafted 10^th item) were subjected to psychometric testing. eTable 1 compares the 10 candidate items with the original 9 items in the pre-existing scale.

The Stanford Professional Fulfillment Index (PFI):⁴⁰ is comprised of the Professional Fulfillment Scale (PFS; 6 items) and Overall Burnout Scale (OBS; 10 items). The OBS combines the 4-item Work Exhaustion and 6-item Interpersonal Disengagement subscales. Each item has five response options: from 0 (“not at all true”) to 4 (“completely true”) for the PFS and 0 (“not at all”) to 4 (“extremely”) for the OBS. Scale scores are derived by averaging scores on constituent items, with averages ranging from 0 to 4. Some studies normalize scores to a 10-point scale, by transforming scores from a 0–4 to a 0–10 range.^25,41–44 However, we applied cut-off thresholds on the 0–4 spectrum in the original validation study.⁴⁰ Thus, respondents whose PFS scores ≥ 3.0 are likely professionally fulfilled. Those whose OBS scores ≥ 1.33 are likely burned out.

The Six-item Physician Perceptions of Autonomy Support (PPAS-6) scale:⁴⁵ assesses physicians’ perceived support towards their clinical autonomy (ie, volition to use one’s best judgment in applying scientific evidence and clinical expertise during patient care). Each PPAS-6 item is rated on a five-point Likert-style spectrum from 1 (“None of the time”) to 5 (“All of the time”). The PPAS-6 is scored by summing up items (after reverse coding a negatively worded “interference” item) such that higher total scores (minimum = 6; maximum = 30) indicate higher autonomy support. One standard deviation above the mean⁴⁶ PPAS-6 score in the original validation study,⁴⁵ rounded to the nearest whole number, was the threshold for “high” ratings on the PPAS-6. Thus, PPAS-6 scores ≥22 (ie, 22–30) indicated perceptions of “high” support towards clinical autonomy; scores between 17 and 21 “moderate” support; and scores ≤17 (ie, between 6 and 16) “low” support.

The Self-Valuation Scale (SVS):⁴⁷ comprises two items assessing deferment of self-care to prioritize work demands (eg, “I put off taking care of my own health due to time pressure”), and two items assessing harsh responses to personal imperfections/errors (eg, “When I made a mistake, I felt more self-condemnation than self-encouragement to learn from the experience”). Items are scored via 5-point Likert response options from 0 (“Never”) to 4 (“Always”). Total SVS scores ≥ 9 are the threshold for moderate-to-high self-valuation, suggesting a respondent is likely to prioritize personal well-being over work and to have a growth mindset. SVS scores <9 indicate low self-valuation suggesting a respondent is likely to defer self-care to prioritize work demands and to respond harshly to personal imperfections or errors.

Organizational retention: Four items assessing intentions to leave the organization in the next 24 months, reduce work hours in the next year, or to voluntarily retire,^48–51 were included. The first item (“What is the likelihood that you will leave your current organization within two years?”) was scored via five response options: 1 “none”, 2 “slight”, 3 “moderate”, 4 “likely”, 5 “definitely”. Two items (“Are you considering leaving or retiring altogether?” and “Are you retiring earlier than you had anticipated retiring?”) had a binary Yes/No response option. A fourth item, “What is the likelihood that you will reduce the number of hours you devote to clinical care over the next 12 months?” had five response options: 5 “none”, 4 “slight”, 3 “moderate”, 2 “likely”, 1 “definitely”.

Peer relationships: Two new items originated by the authors solicited respondents’ self-reported connectedness to peers (“I feel connected to my peers at work”) and respect by peers (“I feel respected by my peers at work”), respectively, via one of five responses: “strongly disagree”, “disagree”, “neither agree nor disagree”, “agree”, or “strongly agree”.

Statistical Analysis Strategy

The study dataset was randomly and equitably split 50:50 into development and validation sub-samples to facilitate a split-sample internal validation strategy. Equitable distribution of contextual variables between derivation and validation subsamples was tested to confirm successful random partitioning. Three alternative formulations of the adapted MLII were compared: a 10-item versus a 9-item versus an 8-item format. Internal consistency of the scale was assessed using ordinal coefficient alpha⁵² and Cronbach’s coefficient alpha.⁵³ Reliability of individual items was tested via inter-item plus item-to-scale polychoric correlations.⁵⁴ The Spearman correlations (ρ) with the OBS assessed divergent validity. Convergent validity was evaluated using correlations (ρ) with the PFS, PPAS-6, SVS, and Peer Connectedness/Respect. Construct validity was tested via single-factor diagonally weighted least squares (WLSMV) confirmatory factor analysis (CFA)⁵⁵ and Samejima’s polytomous graded response item response theory (IRT)⁵⁶ models. Statistical analyses were performed using SAS version 9.4 (SAS Inc., Cary, NC), Mplus^® version 8.6 (Muthen & Muthen, Los Angeles, CA), IBM SPSS^® Statistics version 29.0.0.0 (IBM Inc., New York, NY), and R version 4.2.1 for Windows (R Development Core Team, Vienna, Austria).

Results

Sample Characteristics

Of 500 eligible physicians, 158 submitted survey responses (response rate = 31.6%). Respondents were predominantly male (76.0%), aged 41 to 65 years (63.3%), and White (43.0%) or Asian (30.4%). A plurality (48.74%) had practiced for ≥15 years. Median (Q1, Q3) annual caseload was 200 (50, 520) patient-care encounters per year. Almost one of three (32.3%) were (non-invasive/interventional) cardiologists, with cardiovascular surgeons (9.5%) and anesthesiologists (9.5%) as the next two most self-reported specialties. Table 1 further outlines the sample characteristics. The 50:50 split-sample randomization distributed most demographics and service attributes equitably between derivation (n = 79) and validation (n = 79) subsamples, except for female gender and the middle age groups (41–50 and 51–64 years). However, psychometric indices were identical between derivation and validation subsamples despite the observed differences in distribution of sexes and middle age groups.

Item-Level Scores and Item/Scale Reliability

Of three candidates (10-item, 9-item, and 8-item) adaptations, the 9-item version was selected due to superior psychometric indices. Specifically, the final 9-item adaptation of the MLII excluded the brand-new candidate item and reframed some of the original items of the pre-existing 9-item MLII. Mean (± standard deviation) scores on nine individual items of the adapted MLII, in the derivation subsample, ranged from a low of 3.43 (1.26) on item 1 (“holds career development conversations with me”) to a high of 4.17 (0.84) on item 4 (“ensures I am treated with respect and dignity”). “Strongly disagree” or “Disagree” responses were less frequently endorsed than “Strongly agree” or “Agree” responses on all items (see eTable 2). Scale reliability coefficients if an item is deleted ranged from 0.950 for item 5 to 0.956 for both items 7 and 9 (see eTable 2). eTable 3 illustrates the inter-item and item-to-scale correlation matrix. Inter-item polychoric correlations (standard errors) ranged from a low of 0.68 (0.06) between items 1 and 7 to a high of 0.90 (0.02) between items 5 and 8, indicating moderate to high item reliability. Item-to-scale Spearman correlation coefficients ranged from 0.80 for item 7 to 0.90 for item 6 (p < 0.0001), indicating high reliability of all items. Standardized Cronbach’s alpha coefficient was 0.958, indicating high internal consistency of the composite scale. eFigure 1 illustrates a polychoric correlation heat map of the 10 candidate items initially considered. eTable 4 compares reliability indexes for 8-, 9-, and 10-item candidate adaptations of the revised MLII among the derivation subsample.

Table 2 illustrates the single-factor CFA of the 9-item adaptation based on the WLSMV estimator. The CFA excellently fit the derivation subsample data (SRMR = 0.035; CFI = 0.999; TLI = 0.997), providing evidence of unidimensionality. Standardized loadings (λ_standardized) for all items exceeded 0.700 (p < 0.001). Items 5, 6, and 8 had the highest, second, and third highest factor loadings (λ_standardized = 0.905, 0.894, and 0.890) plus proportions of variance in item scores (R² = 0.819, 0.800, and 0.791) accounted for by the latent factor. Items 9 and 4 had the lowest and second lowest factor loadings (λ_standardized = 0.777 and 0.779) and R² values (0.604, 0.606), respectively. eTable 5 compares CFA goodness-of-fit indexes among the derivation subsample for 8-, 9-, and 10-item candidate adaptations.

Table 2 Diagonal Weighted Least Squares (DWLS) Confirmatory Factor Analysis – Item Loadings and Global Fit Indices

Calibration with the Unidimensional Graded Response Model

Parameter estimates from the unidimensional GRM of the 9-item adaptation are listed in Table 3. Items that more efficiently discriminate among respondents’ leadership ratings have higher/steeper slope (α) parameters. Conventionally, slopes of 0.65–1.34 indicate “moderate”, 1.35–1.75 “high”, and >1.76 “very high” discrimination.⁵⁷ Items 7 and 9 had “high” discrimination; the other seven items “very high” discrimination. Items 5, 2, and 8 most efficiently discriminate between respondents’ ratings of their leaders. Each threshold or difficulty (Ь) parameter is the point at which the probability of respondents endorsing a specific response versus another (eg, “strongly disagree” vs “disagree”) is approximately equal (50:50). Higher Ь values indicate more difficult response options for respondents to endorse. Response category thresholds ranged from −2.570 for Ь1 on item 4 to 0.796 for Ь4 on item 1. Item-level goodness-of-fit was assessed by the generalized S-∑² index, which indicated good overall fit (ie, p ≥ 0.001) for all items, with no item showing poor fit (ie, p < 0.001). Response option characteristic curves in Figure 1 show that respondents endorsed a wide spectrum of responses on all items of the adapted MLII. Thus, the scale validly captures a diverse range of respondents’ ratings of their leaders. Figure 2 shows information function curve plots for individual items and the adapted measure as a whole. Items 5, 8, and 2 captured the highest amount of psychometric information across the entire breadth of variability in leaders’ ratings. Items 7, 9, 4, and 1, in contrast, had the flattest information curves. eFigure 2A and eFigure 2B depict item characteristic and item information curve plots for the 10^th candidate item. eFigure 3 compares test information curve plots for 8-, 9-, and 10-item candidate adaptations.

Table 3 Unidimensional Graded Response Model of the Nine-Item Adaptation of the Revised MLII in the Derivation Subsample – Item Scalability, Slope (Discrimination), and Response Category Threshold (Difficulty) Parameters

Figure 1 Unidimensional Graded Response IRT Model of the Nine-item Adaptation of the Revised MLII – Item Characteristic Curve Plots.

Figure 2 (A) Unidimensional Graded Response IRT Model - Item Information Curve Plots.

Figure 2 (B) Unidimensional Graded Response IRT Model – Test Information Curve Plot.

Convergent and Divergent Validity

A strong positive unadjusted association (ρ = 0.567; p < 0.001) was observed between leadership behavior and perceived autonomy support. The adapted MLII had moderate positive associations with professional fulfillment (ρ = 0.470; p < 0.0001), respect from peers (ρ = 0.496; p < 0.0001), connectedness to peers (ρ = 0.411; p < 0.0001), and a moderate negative association (ρ = −0.303; p = 0.0004) with burnout. A small positive association with self-valuation (ρ = 0.286; p = 0.0009) was observed (see eTable 6).

Discussion

We tested the psychometric validity and reliability of an adapted version of the MLII designed to assess ratings of leaders’ behaviors by physicians embedded in organizations with a multiform, flexible leadership structure. We confirmed the unidimensional factor structure of the nine-item adaptation via CTT and IRT analyses. Its construct validity, convergent and divergent validity, and internal consistency reliability satisfied established criteria.

This study yields evidence that the adapted MLII validly and reliably assesses leadership behaviors experienced by physicians who are neither exclusively supported nor supervised by a single direct-report leader. Furthermore, perceptions of leadership support positively correlated with professional fulfillment, perceived autonomy, self-valuation, and connectedness to peers. Lower scores on the adapted measure correlated with higher levels of burnout. By standard criteria,⁵⁸ correlation coefficients were mostly moderate to high.

Our findings replicate studies of the standard 9-item MLII designed for use in settings with traditional, hierarchical leadership structures. Mete et al found Pearson correlation coefficients (r) of –0.34 with overall burnout and 0.44 with professional fulfillment.¹⁵ Dyrbye et al observed correlations (r) of –0.247 with burnout and 0.444 with satisfaction with one’s organization.¹³ Shanafelt et al found a correlation (r) of 0.53 with values alignment at the work unit level.²⁵ Kang et al observed correlations (r) of 0.72 with psychological safety, 0.67 with excellence/innovation, 0.48 with engagement, and 0.44 with well-being.⁵⁹ Likewise, the reliability coefficients are consistent with studies of previous MLII versions.⁵⁹

Despite a proven association of leadership behaviors with clinicians’ well-being plus values alignment with their organization in settings with traditional, hierarchical leadership structures,^{12–15,25,26} organizations with matrixed leadership reporting structures justifiably question the applicability of such findings to their contexts. Our study demonstrates that leadership behavior remains an important driver of burnout and professional fulfillment even for physicians embedded in settings with no singularly exclusive direct-report leader. This emphasizes the importance of assessing, developing, and fostering well-being centered leadership in such organizations.

One model posits that well-being centered leadership has, at its core, three elements: (1) genuine demonstration of caring about the individuals they lead, (2) cultivation and nurturing of individual relationships and interrelationships among team members, and (3) inspiring work-unit level change by fostering creativity and autonomy as well as supporting change efforts.²⁴ Leadership training programs aimed at teaching skills within these foundational domains may be insufficient to generate long-term improvement. Factors such as leaders’ personality traits, unique needs and expertise, plus the organization’s supportive structures and processes must also be considered to effectively optimize performance.⁶⁰

Limitations and Strengths

This study has limitations. Respondents were from a single healthcare system, likely limiting generalizability. Although comparable to many physician studies,^61,62 our response rate underperforms averages for online surveys of specialist physicians.^63,64 As a sensitivity analysis, we tested CTT and IRT models on an expanded simulated dataset generated via 100 multiple imputations⁶⁵ of the respondents’ sample and observed identical psychometric indexes. The expanded, simulated dataset was generated by using a multiple imputation method to draw an unrestricted random sample from the study dataset 100 successive times with replacement. Thus, the modest sample size was likely not a significant threat to statistical conclusion validity. Authors had no data on non-respondents and could not quantify non-response bias. However, studies conducting robust analyses of survey non-responders show that respondents typically are representative of target subpopulations.⁶⁶ The cross-sectional nature of the study precluded test–retest reliability assessment. Additionally, acquiescence response bias was not assessed in this study. Notable strengths of the study were the robust validity and reliability indices, plus the split-sample internal validation strategy that minimized overfitting.

Implications of the Study

Our findings imply that physicians in organizations with matrixed leadership reporting structures receive “well-being centered” leadership support from diverse sources (eg, direct report leaders, indirect leaders, professional colleagues, and peer groups) and that this multi-sourced support is associated with professional fulfillment and burnout levels.⁶⁷ Future studies might extend the single-factor, uni-dimensional model via a multi-dimensional conceptual framework that unearths distinct (eg, emotional, tangible, and informational)^68,69 domains, not just the “overall” or composite construct, of leadership support. In addition, this study offers a measure that can help identify individuals or workgroups experiencing low levels of well-being focused leadership support. Tailored interventions to improve support can then be developed and implemented.

Conclusion

An adapted version of the MLII validly and reliably assesses well-being centered leadership support in organizations with matrixed leadership reporting structures not dependent on a single direct-report leader. The adapted measure’s validity and reliability indices resemble those of the traditional MLII designed for settings with exclusive, direct-report leaders. Scores on the adapted measure correlate negatively with burnout and positively with professional fulfillment indicating that wellness-centered leadership behaviors are important both for systems with hierarchical leadership structures and matrixed leadership reporting structures. Prior to this study, no equivalent measure had been validated for use among the growing number of healthcare systems with matrixed leadership reporting structures.