Patient Perceived Impact of Outpatient Group Substance Use Disorder Treatment in a Hybrid Model or Virtual-Only Model Relative to In-Person Delivery

Introduction

The COVID-19 pandemic propelled a dramatic transition to telehealth across medicine, including in the treatment of substance use disorders (SUD).¹ The percentage of facilities providing telehealth treatment of SUDs more than doubled in 5 years, from 25.7% in 2015 to 58.6% by 2020, the first year of the pandemic.² Considerable research has shown telehealth’s efficacy in the treatment of opioid, nicotine, and alcohol use disorders, while it is less studied in SUDs without medication for addiction treatment (MAT).³

Benefits of telehealth include its potential to narrow the SUD treatment gap.³ Within addiction treatment, a small portion of individuals ever receive an evidence-based intervention such as psychotherapy or MAT.⁴ Rural populations are disproportionately impacted, with lower rates of overall healthcare utilization.⁵ Telehealth can mitigate transportation issues, decrease total time associated with appointments, and may help minimize treatment-related stigma.^6,7 For people who initiated medication for opioid use disorder (MOUD) treatment during the COVID-19 pandemic, receipt of telehealth services was significantly associated with improved treatment retention and a reduced risk for medical treatment of overdose.⁸

Patient satisfaction with telehealth has largely been positive, with high levels of acceptance and overall feasibility reported.^9,10 Equivalent or greater preference for telehealth services within SUD treatment has been described,^11,12 which is consistent with improved retention rates with telehealth utilization.³ While there may be technical issues at telehealth treatment onset,¹³ these typically resolve with patient experience.¹¹

Group therapy has been a longstanding component of SUD treatment. Transitioning from in-person to online-only may have differential effects on group dynamics. Previous literature comparing in-person to virtual telehealth on ratings of therapeutic alliance (TA), or the patient’s connection with a group facilitator/therapist, has been mixed. Some findings report similar rates of TA,^10,14 while others note a slowed growth.¹⁵ TA has consistently been found to be a significant positive factor across psychotherapeutic interventions.¹⁶ As such, its ability to be achieved and maintained is an important consideration when designing telehealth delivery methods.

Group cohesion, or the level of connection between members of the group with each other and to the group overall, is another relevant aspect of group treatment.¹⁷ Participants may respond robustly to others with shared experiences, learning from each other’s successes and barriers. While available research is limited, previous findings have reported lower rates of cohesion for virtual groups relative to in-person.^12,14

Hybrid groups, or a combination of patients in-person with the therapist while others join virtually, has been essentially unstudied within group therapy treatments of SUDs. We were unable to find any previous reports comparing patients’ perceptions of hybrid groups to exclusively in-person or online. We previously published treatment completion differences between group models.¹⁸ We sought to build upon this work by surveying program participants to incorporate patient self-report data to the literature.

Materials and Methods

This study was reviewed by the Mayo Clinic Institutional Review Board for human research and granted approval. Patients provided informed consent. Data was deidentified. Patient responses had no impact on their care or experience. This study complies with the Declaration of Helsinki.

Participants

Participants (n = 1037) were individuals with a Diagnostic and Statistical Manual of Mental Disorders fifth edition SUD¹⁹ enrolled in outpatient group-based treatment through one of seven programs affiliated with Mayo Clinic Health System in southern Minnesota (median population <30,000), treating a primarily rural population. Programming is delivered 9 hours weekly in a group-based format (8–12 participants) for approximately 3 months and is abstinence-based. Participants are allowed and recommended to take United States Food & Drug Administration (FDA) approved MAT with providers available if clinically appropriate. Treatment completion is contingent upon sustained abstinence, fulfillment of assigned tasks/homework, and regular engagement in group sessions. Program referral may occur through county social workers, other treatment programs, drug/treatment courts and self-referral. Inclusion criteria were 18 years of age or older, a documented history of a SUD, and the ability to participate in outpatient treatment. Exclusion criteria was if a patient had been previously blocked by the health system for treatment due to repeated acts of violence or similar behavior; of note, these treatment programs have low barriers and seek to accept all applicants.

Survey Design and Collection

As previously described,¹⁸ the COVID-19 pandemic led to a change in treatment delivery. Starting June 1^st, 2020, all groups were made virtual, which continued until June 30^th, 2021. Starting July 1^st, 2021, groups transitioned to either being all virtual, all in-person, or where the group counselor was in-person with part of the group while the remaining group members joined virtually (referred to herein as the hybrid group). Our pilot results suggested that virtual attendance may positively augment in-person groups, as opposed to an exclusively virtual approach. We sought to build upon this with patient reported survey data.

The survey began January 1^st, 2022. The program Qualtrics was used. Response was entirely voluntary, and patients could complete part or none of the survey, with partial results used for those questions answered. For in-person participants, a QR code was presented by the group counselor and participants used their phone’s QR reader to view and take the survey. For virtual participants, a link was available in the online chat feature that allowed for survey completion. Each participant was asked to complete the survey one time. To limit any possible repeat surveys, information acquired with the survey (IP address, GPS location) was used by the Mayo Clinic Survey Department to discard repeat surveys before they were sent to the research team. Survey design was a Likert scale. Questions were listed as a continuum of responses “Strongly Disagree”, “Disagree”, “Neutral”, “Agree”, and “Strongly Agree” except for length of treatment, which had answers of “First month of treatment”, “Second month of treatment”, “Third month of treatment”, and “More than 3 months of treatment”.

Statistical Analysis

Data was combined from the 7 IOP sites, with samples ranging from 73 to 186 participants per site. Analysis of variance (ANOVA) evaluated age differences across treatment groups. Analysis of covariance (ANCOVA) tested for differences among treatment groups. Separate models were used for each survey question, where the dependent variable was the survey response, the test of interest being treatment group-type, with covariates of length of stay and age. The interaction term between treatment group and age was evaluated, however was marginally significant for one question, and is described further in the results section. IOP site and the interaction term between cohort site and length of treatment were non-significant and subsequently dropped from all models.

ANCOVA models produced model estimates (adjusted differences) among treatment groups for each survey question, with the in-person group as the reference group. These estimates, along with model-based SD (from the residual error), were used to calculate the Cohen’s d effect size. Effect size cut-offs used Cohen’s standard small, medium, and large effects at values of 0.2, 0.5 and 0.8, respectively.²⁰

Total survey response rate was estimated based on available data for the first 8 months of 2022. Historically, different treatment sites kept inconsistent records on rates of admission and discharge. For the pilot project preceding this survey,¹⁸ we had accurate records of those who had participated in treatment over a certain time period. We therefore compared the total number of survey responses we had from the start of the survey (January 1^st, 2022) to the end of the pilot project (July 31^st, 2022), which provided an 8-month window. While the corresponding analysis went farther than the end of the pilot project, this provided an approximation of a survey response rate.

All analyses were performed using R 4.2.1²¹ with R packages car, dplyr, psych, purr and tibble.

Missing Data

The initial dataset consisted of 1037 participants. Age was missing for 23.9% of the sample, and as age was significant in multiple ANCOVA analyses, it was included for all models. This decision eliminated use of participants who did not have age available, leaving a remaining usable sample of 789 participants. For this narrowed sample, the individual survey questions had a missing data rate between 1.9% and 10.4%, giving a range of used participant responses between 707 and 774 per question (sample sizes available in Table 1).

Table 1 Survey Question Responses

Description of Technology

The technology used to create the hybrid groups has been previously described.¹⁸ In brief, we utilized a combination of cameras and large television monitors to attempt to create an immersive environment for both patients and counselors. The KanDao Meeting 360 All-in-One Conference Video Camera was in the center of the room and has voice-activated technology that places the focus of the camera onto whomever is talking in the room. Their image is then cast on the screens in the room and becomes the prominent available image for those joining virtually. Zoom for Healthcare was used to create a virtual environment for patients to join, with the group in the room also joining as a collective participant. Counselors emphasized participant engagement, working to keep both those in the room and those who were joining virtually, participating in group discussions.

Results

Age was not significantly different across treatment groups, with respective group means for in-person, virtual and hybrid of 40.4 (SD = 10.4), 36.8 (SD = 11.6) and 38.2 (SD = 11.3) (p = 0.127). Table 1 shows each survey question, along with its corresponding sample size, mean and standard deviation for responses. Respondents were overall favorable, with the majority of survey question means being near or above a 4, or the “Agree” response.

Table 2 displays the ANCOVA analysis results, comparing the hybrid and virtual groups to the in-person group. For the hybrid groups, there were no significant differences noted with survey responses relative to in-person. The corresponding effects sizes for each question were also small, with the largest absolute value of the 95% confidence interval being 0.44.

Table 2 Hybrid and Virtual Groups Relative to the in-Person Group

When comparing virtual to in-person, there were significant differences noted. Based on the model adjusted differences, virtual was rated as significantly worse than in-person on 6 of the 8 questions. Reported connection with both peers and the group counselor, along with beliefs that the treatment program helped maintain sobriety and gain insight into one’s addiction, were all rated worse. Virtual group survey respondents were also significantly lower in terms of recommending treatment to others and overall ease of participating in treatment relative to in-person. The 95% confidence intervals for these approached or surpassed threshold for moderate effect (see Table 2). Visual comparison of effect sizes for being in the virtual or hybrid groups relative to in-person groups can be seen in Figure 1.

Figure 1 Effect size of Hybrid or Virtual care relative to In-Person. Notes: Cohens’ d effect size with 95% confidence intervals were calculated based on survey responses from adults in intensive outpatient substance use disorder group treatment. Participants joining exclusively in-person were the reference group for those joining exclusively virtually (Virtual) and those in groups where participants joined both in-person and virtually from home (Hybrid). Effect sizes were calculated from analysis of covariance models for each survey question.

Respondents’ age did not differ significantly across the groups, however as seen in Table 3, age was a significant factor in 4 of the survey questions. For each corresponding ANCOVA model for those 4 questions, age was significantly correlated with a higher degree of positivity in survey responses. Questions related to group cohesion, connection to the counselor, insight gained into one’s addiction and treatment having a positive impact on sobriety all had significantly higher evaluations with increasing age. These increases were minimal however, such as for group cohesion, model estimate was 0.012 with p < 0.001. Stated differently, for every increasing year of age, group cohesion significantly increased by 0.012. We focused less on these values due to the likely small clinical significance of this, with more importance that increasing age was not found to be a negative factor.

Table 3 Impact of Age on Survey Questions

To evaluate if the association of age and responses were affected by group membership, we evaluated the interaction of treatment group and age. Across all survey questions, this interaction was marginally significant for only one question (p = 0.044). This level of significance would not have survived a Bonferroni correction, and the interaction term was subsequently dropped from all models. Thus, the correlation of age and response was considered constant across the groups.

When removing age from all models, which allowed for utilization of all data, the overall results varied minimally from when controlling for age. We therefore did not see evidence for potential bias from those who did not have age available.

Estimation of Survey Response Rate

Combining data from the pilot paper¹⁸ of this project and the current analysis, there were 546 patients who spent at least 1 calendar day in treatment from January 1^st, 2022, to July 31^st, 2022. A total of 414 unique individuals opened the survey, however, were able to exit the survey at any point. This provided a response rate of 75.8% for that 8-month period. Rates of missing data are commented on in the above Methods section.

Discussion

Survey results were consistent with hypotheses generated from the preceding pilot project: respondents favor having the ability to participate in-person. A novel finding, however, was that depending on how the virtual component was incorporated, this significantly impacted patient responses. When all group members were virtual, respondents overall review of the program was significantly worse relative to groups where all members joined in-person. This differed from hybrid groups, whose survey results were not significantly different from those whose groups attended exclusively in-person.

We have previously shown that during the first two years of the COVID-19 pandemic, group attendees who participated in a hybrid group format had significantly higher odds of completed treatment when compared to virtual-only participants.¹⁸ These current findings build upon those results in two keys ways: 1) the incorporation of participant experience, and 2) the addition of an exclusively in-person format, allowing for an improved ability to identify differences between treatment delivery modalities.

The overall effect size of choosing a hybrid group relative to exclusively in-person appears to be small. The largest absolute value of a survey question was 0.13, which is below the cut-off for a small effect. When including the 95% CI’s, these remain below the cut-off for a moderate effect. Therefore, while we cannot claim that a hybrid group is non-inferior to an in-person group, the effect of choosing a hybrid group is at least not a moderate or large effect. This differs from exclusively virtual, where effect size estimates exceeded the small effect size cut-off, and the 95% CI exceeded the moderate cut-off. This provides some evidence that choosing a completely virtual group can have at least a small negative effect.

Potential impacts to TA and group cohesion are important considerations when incorporating a virtual treatment dimension, with previous literature showing mixed findings. In a cohort of 291 individuals aged 18–30 primarily diagnosed with mood and anxiety disorders, growth of TA was slower for participants receiving virtual treatment relative to those in-person.¹⁵ In a small cohort of patients (n = 35) with mood disorders engaged in dialectical behavioral therapy, Lopez et al reported no difference in TA for online versus in-person respondents.¹⁴ For this same sample, while TA was equivalent, group cohesion was significantly lower for online respondents.¹⁴ In a survey of 58 patients with a history of a SUD, 16 reported they found it more challenging to connect with group members.¹²

Our findings, which utilized a larger sample of patients with a SUD, showed that hybrid group respondents reported equivalent measures of TA and cohesion as in-person respondents, whereas virtual-only respondents reported significantly lower measures of TA and cohesion with a negative effect size of small to moderate when compared to in-person respondents. These findings suggest that there may be a moderating effect of having some participants in-person with the counselor that helps support group and treatment dynamics. Alternatively, patients may also prefer the flexibility of attending either in-person or virtually.

Respondents overall rated the program as easy to schedule and easy to attend their appointments regardless of group type (data not shown). There were also uniformly high rates of satisfaction with the respondent’s counselors and treatment program (Table 1). Patient satisfaction within a SUD treatment-seeking population has been previously positively correlated with higher rates of 1-year abstinence,²² providing strength for its relevance as a treatment metric. While our virtual-only groups overall found access to be easy, rural telehealth access is a concern. Broadband internet availability is substantially lower in rural areas relative to metropolitan, which may limit some patients’ opportunities to participate in non-exclusively in-person treatment modalities.²³ Digital access availability is also worse for those 100% below the federal poverty line and over 85 years of age.²⁴

Ability to successfully participate in telehealth services may correlate with age, however, technology literacy is also an important factor.^25,26 Previous research has supported the notion that patients become more comfortable with telehealth with repeated exposure.¹¹ Before COVID-19, there was support for increased use of telehealth in geriatrics,²⁷ and more recently, a scoping review found higher use of telehealth in neuropsychiatric services relative to other areas of geriatrics.²⁸ Our findings showed no significant impact of age on effect sizes for treatment selection. Additionally, in models where age was significant, increasing age correlated with more positively rated responses.

The most exciting finding of this study is that there was no objective data that indicates using a hybrid group format was worse than exclusively in-person. Providing increased treatment options makes sense as an overall positive. The typical implementation of telehealth, however, seems to be either all in-person or all virtual attendees. This all-or-nothing paradigm understandably simplifies the technology needed but may not be the best option.

To incorporate this successfully, there does need to be thoughtful implementation of technology. As previously described,¹⁸ we feel that the immersive environment created by centrally placed cameras, multiple video screens, and attention paid by the counselors to focus on overall group interaction between virtual and in-room attendees are all critical components that affect program success. For certain treatment groups that may meet weekly or more frequent, having the opportunity to attend some virtually also provides flexibility for the patient, and therefore may lead to increased attendance. This may be especially important for rural populations, who typically have a greater distance to travel with fewer treatment options.

This study does have limitations. The one-time survey created a cross-sectional analysis of anonymous patients who were currently enrolled in treatment at differing lengths of time. Therefore, it is unknown whether these findings would hold true several months post-treatment completion. Models did control for current treatment length, providing some context for how responses may change throughout treatment, however this differs from long-term outcomes. Generalizability may also be an issue. Participant response was entirely voluntary, which may bias results and also does not capture every individual’s opinion. We serve a primarily rural population, and all participants had a history of a SUD. Deidentification prevented controlling for SUD type, severity or comorbid diagnoses, which negatively impacts generalizability. While we feel the technology used could be employed for treatment of multiple diagnoses, it is unknown if patients with non-SUDs or in metropolitan settings would have different experiences. As patients were deidentified, including their sex, it is unknown if there was a significant sex difference in survey responses.

Additional limitations include that treatment group selection was voluntary; therefore, we did not have the benefit of randomization to theoretically evenly distribute unidentified potential confounders. It is possible that out of the hybrid group, attendees that primarily joined in-person were the responders, making that cohort like the exclusively in-person group. While we do not have data to absolutely refute this, the online-only group respondent sample sizes were the largest of the 3 cohorts, suggesting that being in the online-only group was not a limiting factor for survey participation. Lastly, we did not correct for multiple hypotheses testing, and therefore results should be considered exploratory with no definitive conclusions able to be drawn.

Conclusion

In this survey-based cross-sectional study of patients with an active SUD currently enrolled in 1 of 7 outpatient SUD treatment centers in rural Minnesota, we saw equivalent overall measures of TA, cohesion, insight gained and impact on sobriety for a hybrid treatment group relative to in-person. When comparing online-only to in-person groups, these findings were all worse, with small to moderate negative effect sizes. Incorporating a hybrid group into treatment paradigms may provide benefits of telehealth such as convenience and flexibility, while avoiding some potential negatives when care is exclusively online. It is unclear the cause of this effect, however having some participants in a shared space with the counselor may help to assist overall flow of communication. Future research looking at long-term effects by treatment modality would help to further characterize the impact of treatment design.