Increasing Acceptability and Outcome Expectancy for Internet-Based Cognitive Behavioral Therapy During the COVID-19 Pandemic

Introduction

Researchers from around the world have documented widespread increases in mental distress since the beginning of the COVID-19 pandemic.^1,2 During this time of increasing need for mental health services, the risk of COVID-19 infection has caused a rapid, large-scale disruption of face-to-face mental health care. Many providers have transitioned to videoconferencing-based telemedicine,³ and researchers have speculated that this transition may spur increased use of digital solutions to systemic health care problems.^4,5 This includes conventional psychotherapy delivered via videoconferencing and e-health mental health interventions that can be completed on one’s own or with relatively brief human support.

Internet and mobile-based digital mental health interventions significantly reduce symptoms for many mental health problems, including depression, anxiety, stress, and substance abuse.^6,7 These programs can be completed independently, that is, “self-guided,” or incorporate support from a therapist or coach. Internet-based mental health programs are effective, but they are widely underutilized,^8,9 and large survey studies conducted before the COVID-19 pandemic have found low levels of acceptability and confidence that these interventions will work.^10–12 However, this may have changed in the context of the COVID-19 pandemic, which dramatically expanded the role of technology in many people’s lives. With millions of people in the United States working from home and switching to technology-mediated forms of communication, there has been a large-scale increase in the use of telemedicine,^13,14 downloads for mental health apps,^15,16 and treatment-seeking for Internet-based mental health programs.¹⁷

Although people’s openness to e-health for mental health care seems to have increased during COVID-19, many who could benefit from these programs may still need persuasion to use them. Video- and text-based treatment rationales designed to improve attitudes toward digital mental health programs have shown promising results.^18–20 These rationales explain how a treatment works and describe the evidence that it is effective. They have been shown to increase acceptability, defined here as general attitudes and beliefs about programs, and outcome expectancy, the belief that a program will be effective. Treatment rationales for e-health mental health programs may be more effective in the context of the COVID-19 pandemic, due to increased distress, low availability of face-to-face treatment, and increased use of technology in day-to-day life. However, no study has examined this possibility by comparing the effects of a treatment rationale for digital mental health programs administered before and during the COVID-19 pandemic.

The current study focuses specifically on the effects of a treatment rationale for Internet-based cognitive behavioral therapy (iCBT), one of the most widely studied forms of e-health for mental health.²¹ The authors re-contacted participants from a large experimental study that examined the effects of a treatment rationale for iCBT before the COVID-19 pandemic.²⁰ Respondents to the follow-up repeated study procedures from the parent study, including the original experimental manipulation of receiving a treatment rationale or not. The authors hypothesized that there would be significant interactions between rationale condition and time point, such that the rationale caused a larger increase in acceptability and outcome expectancy for iCBT when administered during the COVID-19 pandemic, as compared with before the pandemic.

Materials and Methods

Procedure

All procedures for this study were approved by the Institutional Review Board at Georgia State University (IRB00000716).

Parent study

Individuals were recruited for a study examining the acceptability of iCBT from June 2018 to September 2019 (N = 662).²⁰ Participants in this parent study were students at a large university in the southeastern United States and adults from the surrounding urban community. Students participated online, whereas community participants were recruited in public places and completed the study on a tablet computer with a research assistant. Inclusion criteria included age 18 or older and ability to read in English. All participants completed a digital survey in which they were randomized to receive a rationale for iCBT or not. The treatment rationale was ∼800 words in length and described iCBT in depth, using persuasion techniques to increase acceptability and outcome expectancy.²² Participants assigned to the no-rationale condition read a definition of iCBT that was 130 words focusing on the difference between therapist-assisted and self-guided iCBT so that the participants could answer questions about both modalities. Participants then completed self-report measures of acceptability and outcome expectancy for self-guided and therapist-assisted iCBT. Participants also completed a measure of current psychopathology. For further details about the parent study, see Molloy et al.²⁰

Follow-up survey

Participants from the parent study were re-contacted by e-mail in May 2020 and invited to participate in a follow-up survey. People who responded read a treatment rationale (or not) according to their original assignment in the parent study and completed the same self-report questionnaires as in the parent study, as well as a measure examining experiences with the COVID-19 pandemic. All participants were offered online gift cards as compensation for completing the follow-up survey.

Participants

Fifty-four participants (8.2% of the original sample) completed the study from May through July 2020. Differences in demographics and dependent variables between those who completed and did not complete the follow-up study were evaluated by using t tests and chi-square analyses. People who completed the study were more likely to be women (chi-square = 6.377, p = 0.012) and rated therapist-assisted iCBT as significantly more acceptable than those who did not (Attitudes toward Psychological Online Interventions Scale; t = 2.497, p = 0.013). There were no significant differences on other demographic characteristics, psychopathology, familiarity with iCBT, acceptability for self-guided iCBT, or outcome expectancy for self-guided or therapist-assisted iCBT (all p’s > 0.05).

Three questions about the treatment rationale were administered as a manipulation check for participants in the rationale condition. Two participants who answered these questions incorrectly and one participant who completed the parent study in less than 5 min were excluded, resulting in 51 participants whose data were used. Demographics (collected in the parent study) are presented in Table 1. Twenty-one participants were originally randomized to the treatment rationale condition, whereas 30 were originally randomized to the brief definition condition.

Measures

Demographics and use of e-health

A 22-item demographics questionnaire was developed for the parent study by using the Standardized Data Set from the Center for Collegiate Mental Health at Penn State University.²³ Participants also reported whether they were currently using an “online mental health or iCBT program” or had ever used one in the past.

Attitudes toward Psychological Online Interventions Scale

The Attitudes toward Psychological Online Interventions Scale (APOI) is a validated measure of attitudes toward digital mental health interventions, with greater scores reflecting more positive attitudes.²⁴ The APOI has demonstrated strong internal consistency in previous research (α = 0.77).²⁴ It was used in the current study as a measure of acceptability for iCBT, defined as cognitive attitudes toward these interventions.

Credibility/Expectancy Questionnaire, Expectancy Subscale

The expectancy subscale of the Credibility/Expectancy Questionnaire (CEQ)²⁵ is composed of three items that evaluate outcome expectancy for psychological interventions, with greater scores reflecting greater expectations of effectiveness. It is widely used in psychological research and has demonstrated high internal consistency (α = 0.79–0.90) and test–retest reliability (r = 0.83).²⁵

Depression Anxiety Stress Scales 21 Item Version

The Depression Anxiety Stress Scales 21 Item Version (DASS-21)²⁶ is a commonly used measure of psychopathology, with individual subscales for depression, anxiety, and stress. It has strong convergent validity with the Beck Anxiety Inventory (r = 0.81) and Beck Depression Inventory (r = 0.74)²⁶ and strong internal consistency for the overall scale (α = 0.93).²⁷

Pandemic Stress Index

A modified version of the Pandemic Stress Index (PSI)²⁸ evaluated participants’ experiences with the pandemic. It included questions about common experiences related to the COVID-19 pandemic, for example, social distancing, losing employment, or contracting COVID-19. It also assessed whether participants had used various forms of telemedicine or e-health to support their physical and mental health during COVID-19.

Statistical analyses

Impact of COVID-19 and use of telemedicine

The frequency of common experiences with the pandemic and use of telemedicine were evaluated using the PSI. A matched-pairs t test was conducted to test for increases in psychopathology (DASS-21 total score) during the pandemic, as compared with before the pandemic.

Preliminary analyses

A pair of two-way within-subjects analyses of variance was used to test for interactions between time point (pre-COVID-19 vs. during COVID-19 pandemic) and type of iCBT (self-guided vs. therapist-assisted iCBT) on the acceptability of and expectations of effectiveness for iCBT. Because there was not a statistically significant two-way interaction for either acceptability or outcome expectancy of iCBT (F (1, 50) = 1.060, p = 0.308; F (1, 50) = 0.516, p = 0.476, respectively), the type of iCBT was collapsed for the main analyses to increase power.

Main analyses

Two two-way mixed analyses of covariance (ANCOVA’s) were used to test for interactions between time point (pre-COVID-19 vs. during COVID-19 pandemic) and treatment rationale condition (yes, no) to test the hypothesis that exposure to a treatment rationale would produce a greater increase in the acceptability and outcome expectancy for iCBT during the pandemic as compared with before the pandemic. Consistent with the parent study, age and baseline psychopathology (DASS-21 score pre-pandemic) were used as covariates due to evidence that they are related to interest in Internet-based mental health treatment.^11,29 The type of iCBT was collapsed for main analyses by taking the sum of APOI and CEQ scores for self-guided and therapist-assisted iCBT, respectively. A Bonferroni correction of α = 0.025 was used for all analyses to minimize Type 1 error for multiple comparisons, as each test was conducted with two dependent variables: acceptability and outcome expectancy for iCBT. All data were analyzed by using SPSS version 25.0.

Results

Missing Data

Across all measures used for the current study’s analyses, there were 11 missing values (0.002% of data). Data were missing completely at random (Little’s MCAR Test, p > 0.05), and missing values were imputed by using expectation maximization.³⁰

Impact of COVID-19 and Use of Telemedicine

See Table 2 for a summary of participants’ experiences during COVID-19. A high proportion of participants reported that their lives had been impacted by the COVID-19 pandemic; the most common experiences included social distancing, following COVID-19-related media, and worrying about friends, family, and others. As shown in Table 3, a majority of participants had not used telemedicine or other electronic resources during COVID-19 to support their physical or mental health.

A matched-pairs t test was used to test for differences in psychopathology before and during the pandemic. DASS-21 total scores recorded before the pandemic (M = 37.99, SD = 25.76) and during the pandemic (M = 37.95, SD = 25.16) were highly correlated, r = 0.593, p < 0.001, and there was no significant difference between them, t = 0.013, p = 0.989.

Main Analyses

Effects of time point, treatment rationale, and their interaction on acceptability and outcome expectancy for iCBT

See Table 4 for results of main analyses. A pair of two-way ANCOVA’s (rationale × time point with age and psychopathology as covariates) tested the hypothesis that receiving a treatment rationale for iCBT (vs. no rationale) would cause a greater increase in acceptability and outcome expectancy for iCBT during the pandemic than before the pandemic. Statistical assumptions were met for two-way ANCOVA, including normality of residuals, homogeneity of variance and regression slopes, and homoscedasticity. For the ANCOVA examining acceptability, there was one residual outlier that significantly affected results.

There was not a significant interaction or main effect of time point for either dependent variable (p’s > 0.033), nor a significant main effect of the experimental condition on outcome expectancy for iCBT (p = 0.668). There was, however, a statistically significant main effect of the experimental condition on the acceptability of iCBT, such that receiving a rationale for iCBT (vs. no rationale) produced greater acceptability of iCBT (p = 0.022). The ANCOVA examining acceptability had one residual outlier (studentized residual = −3.12) and was re-run with this case removed. When the residual outlier case was removed, the main effect of rationale fell to non-significance (p = 0.045). This approach did not change any other aspects of the results.

Discussion

This is the first longitudinal study to compare the effects of a treatment rationale for e-health mental health interventions before and during the COVID-19 pandemic. Consistent with previous research, the treatment rationale used in the current study significantly increased the acceptability for iCBT with a medium to large effect size. These findings replicate previous studies^18,31 and indicate that treatment rationales can increase the acceptability of iCBT. However, the treatment rationale was not shown to be more effective in the context of COVID-19 and was not shown to affect participants’ expectations that iCBT would be effective.

It is possible that the treatment rationale may have impacted aspects of acceptability that are distinct from participants’ expectations that iCBT will be effective. There are several content areas within the measure of acceptability used in this study that could account for this, including perceptions that technology-based mental health interventions are risky (e.g., by increasing isolation), concerns about maintaining motivation or learning skills in the absence of a therapist, and potential benefits of greater confidentiality and reduced stigma that come with using an online mental health program. Theoretical models of technology adoption, such as the Unified Theory of Acceptance and Use of Technology,³² propose a range of constructs, including outcome expectancy, that impact decision making about whether to use interventions such as iCBT. Researchers should draw from these models and continue to examine the ways that acceptability-facilitating interventions such as treatment rationales might improve specific dimensions of iCBT acceptability for individuals who have been impacted by COVID-19.

Health care providers may find treatment rationales for iCBT to be a useful decisional aid for patients who are considering a variety of mental health care options. Providing upfront education about iCBT for treatment-seeking individuals with mild to moderate symptoms may lead them to choose iCBT in lieu of face-to-face care. This is consistent with the goals of shared decision making, a framework used in many health care settings to collaborate with patients and promote their autonomy when choosing their course of treatment.³³ If significant numbers of patients choose iCBT, this would conserve providers’ time for patients with more severe symptoms, a critical goal given the long-standing problems with mental health care access in the United States that have been exacerbated by increased demand during COVID-19.³⁴

Surprisingly, although nearly all of the current study’s participants reported that their lives were affected by the pandemic, there were no significant differences in psychopathology before and during the pandemic, which perhaps helps explain why a minority of participants had used any digital resources for mental health care (31.4%) or online programs such as iCBT (15.7%). These results are inconsistent with studies showing that people have experienced increased anxiety and depression due to COVID-19¹ and sought telehealth services at increased rates, including iCBT specifically.¹⁷ Future studies with participants who experienced a need for health care and significant disruption in access to face-to-face services during COVID-19 may find that these individuals have become more responsive to treatment rationales for e-health, even if this change is not evident in non-treatment seeking samples.

iCBT may particularly benefit communities that have been disproportionately impacted by COVID-19 and have lower access to health care, such as Black Americans,³⁵ people in rural communities,³⁶ and people experiencing homelessness.³⁷ Researching the types of experiences that may increase people’s responsiveness to acceptability-facilitating interventions for iCBT, including experiences with COVID-19 and telemedicine, is an important way to promote health equity by increasing access to care. The current study found that a treatment rationale significantly improved acceptability for iCBT in a racially diverse sample of adults. However, the sample was also relatively young, predominantly female, and recruited from an urban area. Future researchers examining this topic should make efforts to recruit diverse samples, study specific vulnerable communities, and report the demographics of their samples.

Limitations

There are several important limitations to this study. It is possible that the effect of the treatment rationale was increased by the fact that participants read it twice—once during the parent study and again at follow-up. Thus, a greater difference between experimental groups during the pandemic as compared with the pre-pandemic (the central hypothesis of this study) could be a cumulative effect of administering the rationale twice. Given the 1- to 2-year gap between time points, the authors believe this is unlikely. However, future studies examining the effects of acceptability-facilitating interventions for iCBT over time should control for this potential source of bias if possible.

Participants who completed this study reported more positive attitudes toward iCBT during the parent study than those who did not. This type of bias, which can result when large numbers of people are invited to participate in a study on e-health interventions and a small proportion of them volunteer, is unfortunately a common problem in this research area (for a discussion of this issue, see Mohr et al.³⁸). Accordingly, inferences should be drawn cautiously from these results. For example, participants may have already been highly responsive to a treatment rationale for iCBT before the pandemic, which could limit changes in the effect of the rationale due to experiences with COVID-19. Although this study was sufficiently powered to detect large effects, the sample size also limited the ability to detect medium or small effects—even the fairly large effect of the treatment rationale on the acceptability for iCBT fell below significance when an outlier was removed. In summary, given the characteristics of our sample and its relatively small size, these findings should be taken as preliminary and replication is needed.

Conclusion

The treatment rationale used in this study significantly increased acceptability for iCBT during the COVID-19 pandemic. However, it was not shown to be more effective during the pandemic as compared with before the pandemic. Continued research is needed to explore the effects of treatment rationales and other acceptability-facilitating interventions for individuals who have been affected by COVID-19. As health care systems expand their use of telemedicine and e-health programs for mental health, this line of research has significant potential to engage greater numbers of patients with these effective and accessible interventions.

Authors’ Contributions

A.M. and P.L.A. devised the project, the main conceptual ideas, and protocol outline. A.M. coordinated data collection for the study, conducted all statistical analyses, and designed the tables. A.M. and P.L.A. contributed to writing the article. P.L.A. supervised the project.

Disclosure Statement

No competing financial interests exist.

Funding Information

The first author was supported by the Heath Resources and Services Administration (HRSA) Training in Integrated Pediatric Psychology Services Fellowship at Georgia State University, Grant No. D40HP19643.