Telemedicine for Acute Conditions During COVID-19: A Nationwide Survey Using Crowdsourcing
Introduction
Telemedicine, or virtual visits, has long been considered a viable alternative to face-to-face medical consultations.1 According to the Centers for Medicare & Medicaid Services,* telemedicine “seeks to improve a patient’s health by permitting two-way, real-time interactive communication between the patient and the physician or practitioner at the distant site.”
A large body of literature shows that telemedicine entails positive outcomes including improving treatment of patient symptoms and disease management.2–7 Patients feel that telemedicine yields high satisfaction, allows good communication, saves time, and is cost-effective.5–10 Telemedicine decreases hospitalization rates and visits to the emergency room.6 Studies also found that telemedicine yields similar results as face-to-face encounters.8,9,11–14 It was even considered to be the major channel for care delivery15; however, its implementation and usage have still been relatively low.16
The advent of the COVID-19 outbreak resulted in a rapid and significant adoption of telemedicine for both chronic and acute conditions.1,17,18 This skyrocketing prevalence is driven by the need of stopping the spread of COVID-19 with many patients using telemedicine for the first time during the pandemic.19 The usage is also driven by the supply of insurance coverage by payers and infrastructure by providers.20–23
However, both the need and the supply may be temporary, and all parties will face a decision about whether to continue adopting telemedicine after the pandemic. Health care providers need to decide whether to keep developing telemedicine to a mature level. For those paying the bill (e.g., insurance companies, self-insured companies, government agencies), the decision is whether to keep reimbursing telemedicine services, and to what extent. There is evidence of growing support by the government for increasing telemedicine to underserved populations. Patients, especially the first timers, need to consider whether to continue using telemedicine to access care.
One of the considerations that influence the decisions of all parties is whether telemedicine can result in desirable outcomes.24 Although its outcomes have been extensively studied in existing literature, the studies are mostly limited to chronic diseases and single patient group.5,25 They mostly focus on clinical (e.g., effectiveness) and patient (e.g., satisfaction) outcomes, but process outcomes (e.g., waiting time) that may affect patient outcomes are seldom reported.
Researchers have called for persisting promotion and use of telemedicine after the crisis.20,26,27 Leveraging the historically large pool of telemedicine users during COVID-19,1 this study aims to understand the outcomes of telemedicine for acute conditions across patient groups using a national sample. The primary outcomes of interest include patient outcomes, that is, satisfaction and continued use intention, and process outcomes, that is, patient waiting time. To understand the future demand, we also examine whether the patient outcomes are affected by demographic characteristics and differ between returning users and first timers. This study contributes to the health informatics literature by providing a current understanding of how patients experience and perceive telemedicine facing acute health conditions using a national sample.
Materials and Methods
Procedure
We conducted a survey to understand the impact of subjects’ demographic characteristics and experience with virtual visits (i.e., waiting time and experience with telemedicine), on patient outcomes including satisfaction and future use intention. The instruments for satisfaction and future use intention were developed based on previously validated instruments with necessary alternation to keep the survey concise and focused.28–33 The survey was programmed in Qualtrics and was conducted via Amazon Mechanical Turk (MTurk), a crowdsourcing platform developed by Amazon,† where Workers (Turkers) complete human intelligence tasks (HITs) assigned by Requesters.
Studies found that MTurk is a rapid and reliable method to survey a convenience sample,34,35 and that Turkers are as attentive to the instructions as, if not more than, participants from other sources.34,36 Our data mainly consist of patient experience, symptoms, and satisfaction/attitudes. Several previous health care studies have collected these types of data using MTurk.
For patient experience, studies asked Turkers to report work/family conflict and their familiarity with specific features of personal health information management systems.37,38 For symptoms or health status, Turkers self-reported their clinical symptoms and status for mental health,39,40 their general health,38 their additive behaviors,41 and their histories and symptoms of attention deficit hyperactivity disorder (ADHD).42 For satisfaction or attitudes, Turkers were asked about their life and work satisfaction,38,39 their patient portal preference,43 and their attitudes and perception.40,44
It is well supported in literature that health-related data collected from MTurk are reliable, valid, and comparable with data collected from other sources. Mortensen and Hughes conducted a review of 35 articles to assess Amazon’s MTurk as a method for collecting data for health care research.35 Conclusions support that in addition to the obvious cost benefits of collecting data via MTurk, the data should also be considered reliable and of high quality.
Good internal reliability, test–retest reliability, and concurrent, convergent, and/or criterion-related validity are found in scales for addictive behaviors and mental health.40,41 Similarly, satisfactory test–retest reliability of demographics, criterion-related validity of self-reported clinical symptoms, and comparable associations with established benchmarks between symptoms and demographics are found in mental health surveys.39
High test–retest reliability and comparable associations among variables are found in a study examining organization and occupational health.38 High internal consistency is found in scales for familiarity with personal health information management.37 Studies concerning surgent attribute importance, and ADHD histories and symptoms also have findings comparable with data collected utilizing other methods.42,44
Studies that require Turkers’ judgment also suggest that MTurk is a viable tool for data collection. For example, a high correlation is found between faculty experts and MTurk crowd workers for evaluating effectiveness of surgical technical skills.45 MTurk responses were found to be a reliable source for developing categories for coding social media discussion regarding diabetes.46 Amazon MTurk has also proven to be an effective method for verifying relationships between biological ontologies.47 Thus, the evidence from prior medical and health care research supports MTurk as a viable tool for data collection.
However, the merits of MTurk data are valid only if problematic responses are identified and excluded, which can be done by using exclusion procedures.34 We take several steps to ensure the quality of our data. First, Turkers must have an HIT approval rate of >95% to be eligible.48 Second, responses must pass all three attention and consistency checks. An instructional manipulation check was used by asking the respondents to provide a particular answer.34 The two consistency indices include two questions for residency country (must be in the United States [U.S.]) and two questions for their chief complaints for the virtual visits (must be an acute condition).34,40,41 Third, responses that were completed unusually fast were excluded.41,49 Lastly, responses that could rarely occur were also excluded.39
Respondents should satisfy all the following inclusion criteria: (1) ≥18 years old, (2) residing in the U.S., (3) used telemedicine for acute conditions after January 2020 when the first case was confirmed in the U.S., and (4) an HIT approval rate of >95% to ensure the quality of the responses.48 Respondents were compensated $1 for each completed survey.
Measurements
Our survey includes four parts.‡ The first part collects respondent characteristics, including age, gender, race, education, residing state, area of living, and daily internet use. The second part includes four questions that capture their virtual visit experience, including wait time to establish an appointment (WaitDays), wait time to confer with the provider once connected online (waiting time in the virtual waiting room) (WaitMinutes), whether they used the virtual visit for the first time, and the chief complaints about virtual visits. The two waiting time questions were inspired by prior research.31 Options for determining chief complaints by patients were also synthesized from previous studies.28,32
The third part builds upon past research adopting six items to assess expressed patient satisfaction levels.28–30 The last part assesses their future use intention utilizing four items from prior telehealth research.33 We consider the two waiting time questions as our process outcome measures, as well as satisfaction and intention as the patient outcome measures. Questions on satisfaction and future use intention were rated on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree).
Analysis
The validity of both instruments (satisfaction and future use intention) was assessed by confirmatory factor analysis. Cronbach’s alpha and construct reliability were calculated for reliability. Bivariate analyses were conducted to assess relationships between respondent characteristics/experience (including waiting time) and patient outcomes (satisfaction and intention) using one-way analyses of variance for categorical characteristics and Pearson’s correlation for continuous characteristics. We further used linear regression models to estimate the relationships between patient outcomes (satisfaction and intention) and respondent characteristics, experience, and waiting time.
Results
We collected 1,727 responses during the period of June 17–29, 2020, but only 1,114 were eligible, completed, and satisfactory in terms of passing all attention checks. We further excluded 55 unreasonable responses (response time <120 s, age >90, waiting time for an appointment >30 days, and waiting time in the virtual waiting room ≥120 min), resulting in n = 1,059 for analysis. The average age of our sample was 36.44. A typical respondent was at age 30–39 (37.1%), male (61.9%), white (71.9%), having college degree (59.6%), using 1–3 h of internet daily (38.8%), and living in urban areas (47.9%) (Table 1).
| n (%) MEAN ± SD | SATISFACTION (1–7) | INTENTION (1–7) | |||
|---|---|---|---|---|---|
| MEAN ± SD PEARSON’S r | p | MEAN ± SD PEARSON’S r | p | ||
| Age | 36.44 ± 10.63 | 0.01 | 0.838 | 0.2 | 0.459 |
| Age groups | 0.036 | 0.044 | |||
| <30 | 316 (29.8) | 5.15 ± 1.24 | 5.08 ± 1.28 | ||
| 30–39 | 393 (37.1) | 5.41 ± 1.19 | 5.34 ± 1.23 | ||
| 40–49 | 211 (19.9) | 5.24 ± 1.15 | 5.13 ± 1.26 | ||
| 50+ | 139 (13.1) | 5.24 ± 1.25 | 5.24 ± 1.29 | ||
| Gender | 0.028 | 0.019 | |||
| Female | 402 (38.0) | 5.40 ± 1.2 | 5.34 ± 1.26 | ||
| Male | 656 (61.9) | 5.20 ± 1.21 | 5.12 ± 1.26 | ||
| Other | 1 (0.1) | 5.83 | 6 | ||
| Race | 0.949 | 0.950 | |||
| Asian | 35 (3.3) | 5.25 ± 1.16 | 5.19 ± 1.2 | ||
| African American | 146 (13.8) | 5.19 ± 1.37 | 5.18 ± 1.35 | ||
| Hispanic | 44 (4.2) | 5.4 ± 1.09 | 5.22 ± 1.43 | ||
| Middle Eastern | 1 (0.1) | 4.67 | 4.5 | ||
| Native American | 56 (5.3) | 5.28 ± 1.18 | 5.17 ± 1.18 | ||
| White (non-Hispanic) | 761 (71.9) | 5.28 ± 1.19 | 5.21 ± 1.25 | ||
| Other | 9 (0.8) | 5.44 ± 1.6 | 5.64 ± 1.32 | ||
| Prefer not to state | 7 (0.7) | 5.45 ± 0.61 | 5.64 ± 0.73 | ||
| Education | 0.001 | 0.022 | |||
| High school or less | 32 (3.0) | 5.90 ± 1.01 | 5.64 ± 1.27 | ||
| Some college | 115 (10.9) | 5.27 ± 1.31 | 5.07 ± 1.62 | ||
| College degree | 631 (59.6) | 5.18 ± 1.23 | 5.15 ± 1.2 | ||
| Master’s degree or more | 275 (26.0) | 5.41 ± 1.11 | 5.34 ± 1.21 | ||
| Prefer not to state | 6 (0.6) | 5.78 ± 0.74 | 5.12 ± 1.7 | ||
| Area | 0.560 | 0.586 | |||
| Rural | 180 (17.0) | 5.20 ± 1.31 | 5.12 ± 1.39 | ||
| Suburban | 361 (34.1) | 5.32 ± 1.18 | 5.21 ± 1.29 | ||
| Urban | 507 (47.9) | 5.27 ± 1.19 | 5.24 ± 1.2 | ||
| Prefer not to state | 11 (1.0) | 5.42 ± 1.08 | 5.32 ± 1.29 | ||
| Daily internet use | <0.001 | <0.001 | |||
| <1 h | 284 (26.8) | 5.00 ± 1.28 | 4.98 ± 1.3 | ||
| 1–3 h | 412 (38.8) | 5.29 ± 1.13 | 5.21 ± 1.16 | ||
| >3 h | 363 (34.2) | 5.48 ± 1.19 | 5.39 ± 1.33 | ||
| First timer? | 0.951 | 0.732 | |||
| Yes | 865 (81.5) | 5.28 ± 1.22 | 5.20 ± 1.29 | ||
| No | 194 (18.3) | 5.27 ± 1.14 | 5.24 ± 1.16 | ||
| Condition | |||||
| COVID-19 related | 394 (37.1) | 5.28 ± 1.22 | 0.889 | 5.24 ± 1.23 | 0.485 |
| Skin problems | 294 (27.7) | 5.23 ± 1.15 | 0.45 | 5.13 ± 1.25 | 0.192 |
| Eye problems | 241 (22.7) | 5.05 ± 1.25 | <0.001 | 5.05 ± 1.25 | 0.025 |
| Digestive symptoms | 209 (19.7) | 5.36 ± 1.24 | 0.283 | 5.3 ± 1.22 | 0.238 |
| Sinus problems | 199 (18.8) | 5.30 ± 1.20 | 0.768 | 5.18 ± 1.28 | 0.775 |
| Urinary problems | 111 (10.5) | 5.24 ± 1.34 | 0.756 | 5.27 ± 1.34 | 0.61 |
| Ear problems | 110 (10.4) | 5.19 ± 1.24 | 0.43 | 5.12 ± 1.3 | 0.43 |
| Musculoskeletal | 72 (6.8) | 5.44 ± 0.98 | 0.245 | 5.37 ± 1.2 | 0.255 |
| Other | 45 (4.2) | 5.53 ± 1.33 | 0.155 | 5.33 ± 1.47 | 0.516 |
| WaitDays | 2.76 ± 4.15 | −0.09 | 0.004 | −0.06 | 0.04 |
| WaitMinutes | 19.44 ± 16.47 | −0.09 | 0.002 | −0.05 | 0.10 |
Eighty-one and a half percent of the respondents used virtual visits for the first time during this COVID-19 pandemic. The majority of them used it for COVID-19-related complaints, including coughing, fever, and sore throat (37.1%). For process outcomes, respondents waited for the appointment for 2.76 days (median = 2 days) on average, and the majority of them waited no more than 2 days (66%). Respondents waited for the providers after being connected online for 19.44 min (median = 15 min) on average, and many of them waited no more than 10 min (45%) (Table 1 and Figs. 1 and 2).
Fig. 1. Distribution of waiting time for the appointment (in days).
Fig. 2. Distribution of waiting time for the provider after being connected online (in minutes).
The results from confirmatory factor analysis, Cronbach’s alpha, and construct reliability confirmed that our 6-item instrument for satisfaction and 4-item instrument for intention were valid and reliable, as all test results were within the recommended ranges (Table 2).50
| RECOMMENDED VALUE | SATISFACTION | INTENTION | |
|---|---|---|---|
| Comparative fit index | ≥0.90 | 0.99 | 0.99 |
| Tucker/Lewis index | ≥0.90 | 0.98 | 0.98 |
| Root mean square error of approximation | ≤0.08 | 0.07 | 0.08 |
| Cronbach’s alpha | ≥0.70 | 0.91 | 0.88 |
| Construct reliability | ≥0.70 | 0.92 | 0.91 |
Overall, respondents reported moderate satisfaction with virtual visits. Specifically, respondents agree that using virtual visits is comfortable (mean 5.34 of 7), acceptable (5.30 of 7), helpful (5.08 of 7), and convenient (5.32 of 7). They also agree that their problems were treated or addressed (5.25 of 7), and they are satisfied with virtual visit overall (5.35 of 7). In general, about 72–80% of the respondents agree with the satisfaction statements (Table 3).
| n = 1,059 | MEAN ± SD | STRONGLY AGREE | AGREE | SOMEWHAT AGREE | NEUTRAL | SOMEWHAT DISAGREE | DISAGREE | STRONGLY DISAGREE |
|---|---|---|---|---|---|---|---|---|
| Satisfaction (score = 1–7) | ||||||||
| I feel comfortable communicating with the clinician using virtual visits. | 5.34 ± 1.39 | 196 (18.5) | 388 (36.6) | 261 (24.6) | 82 (7.7) | 79 (7.5) | 37 (3.5) | 16 (1.5) |
| Virtual visits are an acceptable way to receive health care services. | 5.3 ± 1.42 | 199 (18.8) | 361 (34.1) | 269 (25.4) | 99 (9.3) | 70 (6.6) | 42 (4.0) | 19 (1.8) |
| Virtual visits are as helpful as office visits. | 5.08 ± 1.51 | 173 (16.3) | 320 (30.2) | 268 (25.3) | 116 (11.0) | 100 (9.4) | 61 (5.8) | 21 (2.0) |
| Virtual visits are more convenient than an office visit. | 5.32 ± 1.52 | 274 (25.9) | 286 (27.0) | 242 (22.9) | 105 (9.9) | 89 (8.4) | 41 (3.9) | 22 (2.1) |
| Virtual visit providers were able to treat or address what was bothering me. | 5.25 ± 1.41 | 195 (18.4) | 339 (32.0) | 273 (25.8) | 120 (11.3) | 74 (7.0) | 40 (3.8) | 18 (1.7) |
| Overall, I am satisfied with virtual visits. | 5.35 ± 1.44 | 241 (22.8) | 339 (32.0) | 237 (22.4) | 109 (10.3) | 79 (7.5) | 37 (3.5) | 17 (1.6) |
| Future Use Intention “After the Covid-19 emergency is over, …” (score = 1–7) | ||||||||
| I intend to use virtual visits if it is difficult to get an office visit at a convenient time to receive care for acute conditions. | 5.32 ± 1.42 | 223 (21.1) | 341 (32.2) | 251 (23.7) | 120 (11.3) | 63 (5.9) | 46 (4.3) | 15 (1.4) |
| I am willing to use virtual visits to complement my traditional care for acute conditions. | 5.29 ± 1.44 | 211 (19.9) | 350 (33.1) | 251 (23.7) | 104 (9.8) | 84 (7.9) | 42 (4.0) | 17 (1.6) |
| I have the intention to use virtual visits routinely to receive care for acute conditions. | 5.1 ± 1.49 | 172 (16.2) | 331 (31.3) | 268 (25.3) | 106 (10.0) | 102 (9.6) | 63 (5.9) | 17 (1.6) |
| I would prefer virtual visits to office visits for acute but nonserious illnesses. | 5.12 ± 1.55 | 203 (19.2) | 301 (28.4) | 260 (24.6) | 119 (11.2) | 90 (8.5) | 59 (5.6) | 27 (2.5) |
Similarly, we observed moderate intention to use virtual visits after the COVID-19 emergency is over. Specifically, respondents agree that they will use virtual visit when it is difficult to get a face-to-face appointment (5.32 of 7), as a supplement to face-to-face appointments (5.29 of 7), routinely (5.10 of 7), and as a preferred approach (5.12 of 7). About 72–77% of the respondents generally agree with the intention statements. While only 16.2% of the patients strongly agree that they will use virtual visits routinely for acute conditions, 19.2% strongly agree that they would prefer virtual visits to office visits for acute but nonserious illnesses (Table 3).
In bivariate analysis between participant characteristics and patient outcomes of virtual visits (Table 1), both mean satisfaction and mean intention were significantly different across groups of age (satisfaction p = 0.036; intention p = 0.044), gender (satisfaction p = 0.028; intention p = 0.019), education (satisfaction p = 0.001; intention p = 0.022), and daily internet use (satisfaction p < 0.001; intention p < 0.001).
In bivariate analysis between participant experience and patient outcomes of virtual visits, significantly lower satisfaction was observed among respondents who visited the doctors virtually for eye problems (p < 0.001), and who experienced longer waiting time for the initial appointment (p = 0.004) and to see providers after being connected online (p = 0.002). Similarly, significantly lower intention was observed among respondents who visited the doctors virtually for eye problems (p = 0.025), and who experienced longer waiting time for the appointment (p = 0.04).
Our multivariable modeling also indicated the same results as the bivariate analysis, except that age was not a significant factor when other variables are controlled (Table 4).
| SATISFACTION | INTENTION | |
|---|---|---|
| Age | −0.00 (0.00), 0.696 | 0.00 (0.00), 0.715 |
| Gender | ||
| Female | 0.18 (0.08), 0.018 | 0.22 (0.08), 0.006 |
| Male [reference] | ||
| Race | ||
| African American [reference] | ||
| White (non-Hispanic) | 0.05 (0.12), 0.684 | −0.01 (0.12), 0.949 |
| Other | 0.07 (0.15), 0.630 | −0.03 (0.15), 0.860 |
| Education | ||
| High school or less | 0.68 (0.18), <0.001 | 0.48 (0.23), 0.033 |
| Some college | −0.03 (0.13), 0.831 | −0.18 (0.16), 0.272 |
| College degree [reference] | ||
| Master’s degree or more | 0.27 (0.08), 0.001 | 0.22 (0.09), 0.013 |
| Area | ||
| Rural [reference] | ||
| Suburban | 0.11 (0.11), 0.331 | 0.08 (0.12), 0.519 |
| Urban | 0.08 (0.11), 0.491 | 0.11 (0.12), 0.339 |
| Daily internet use | ||
| <1 h [reference] | ||
| 1–3 h | 0.31 (0.09), <0.001 | 0.24 (0.10), 0.010 |
| >3 h | 0.45 (0.10), <0.001 | 0.42 (0.10), <0.001 |
| First timer? | ||
| Yes | −0.00 (0.09), 0.978 | −0.04 (0.09), 0.672 |
| No [reference] | ||
| Condition | ||
| Eye problems | −0.25 (0.09), 0.006 | −0.19 (0.09), 0.039 |
| Other [reference] | ||
| WaitDays | −0.02 (0.01), 0.035 | −0.01 (0.01), 0.141 |
| WaitMinutes | −0.00 (0.00), 0.095 | −0.00 (0.00), 0.383 |
In particular, compared with males, females had significantly higher satisfaction and intention (satisfaction B = 0.18, p = 0.018; intention B = 0.22, p = 0.006). When compared with respondents with a college degree, respondents with high school or less (satisfaction B = 0.68, p < 0.001; intention B = 0.48, p = 0.033) or a master’s degree or more (satisfaction B = 0.27, p = 0.001; intention B = 0.22, p = 0.013) reported significantly higher satisfaction and intention. Compared with respondents who used the internet less than 1 h daily, respondents who used 1–3 h (satisfaction B = 0.31, p < 0.001; intention B = 0.24, p = 0.010) or over 3 h (satisfaction B = 0.45, p < 0.001; intention B = 0.42, p < 0.001) reported significantly higher satisfaction and intention.
In conclusion, the more the daily internet usage, the better the patient outcomes. Compared with respondents who used virtual visits for other problems, respondents who visited for eye problems had significantly lower satisfaction and intention (satisfaction B = −0.25, p = 0.006; intention B = −0.19, p = 0.039). Waiting time for appointments had a significant and negative association with satisfaction (B = −0.02, p = 0.035), but this negative association was not observed for intention. Waiting time for providers after being connected online did not have any effect on patient outcomes of virtual visits.
Discussion
Our primary results indicated that the median waiting time for the virtual visit appointment and for the providers was 2 days and 15 min, respectively. The majority of the respondents waited no more than 2 days for the appointment (66%) and no more than 10 min in the virtual waiting room (45%). This is shorter than the reported 20 min of average waiting time in the physical waiting room in the article by Polinski et al.31
For patient outcomes of telemedicine, overall, our respondents reported moderate satisfaction and future use intention, and about 70–80% of the respondents were positive about both outcomes. This dominant positive attitude among the respondents is similar to the findings of existing studies.28,30,31 While only 16.2% of the patients strongly agree that they will use virtual visits routinely for acute conditions, 19.2% strongly agree that they would prefer virtual visits to office visits for acute but nonserious illnesses. Overall, about 72–77% of the respondents reported that they have intention to use virtual visits after COVID-19 is over.
Respondents who were more satisfied and more likely to use virtual visits in the future were female, on the ends of the education spectrum (high school or less and master’s degree or more), and make greater use of the internet. Our finding that females have a positive attitude toward telemedicine is consistent with literature.31 In fact, females were found to account for the majority of telemedicine services.17,28,31 The positive influence of internet use on the attitudes toward technology-aided health communications is also consistent with literature.51 Noticeably, telemedicine seems not to work across all acute conditions, as respondents seeking service for eye problems reported significantly lower satisfaction and future use intention than other patients.
Our results also show that long patient waiting time negatively impacted satisfaction, in line with the literature.52 However, it did not have a significant impact on future use intention, probably because the waiting time is still shorter than that of in-person visits.
Eighty one and a half percent of our sample used virtual visits for the first time during this COVID-19 pandemic, much more than the 50% found in the article by Polinski et al. during a 9-month noncrisis period.31 The pandemic has necessitated the use of telemedicine by patients who otherwise may have never considered the treatment mode as an option.
If we consider these first timers as nonadopters before this pandemic, we would expect that their satisfaction and future use intention would have been lower than that of the early adopters, who had experience using virtual visit before this pandemic. However, our survey results indicate that patient outcomes were not different between these two groups of patients. If we consider nonadopter’s usage during COVID-19 as trials, our results suggest that the trials were successful and may lead to future adoption. Prior studies also found that most of the patients who used telemedicine service were willing to use it again,28,30,31 demonstrating the general positive effect of trials. This COVID-19 situation provides a natural opportunity for the first timers to try the approach, and a high level of retention can be expected, as 72–77% of our respondents reported positive future use intention.
For patients living in different areas (i.e., urban, suburban, and rural), one would expect that patients in rural areas would have higher intention to use it in the future, as the technology improves care accessibility. However, we did not observe any difference in patient outcomes among these patient groups. This may be attributed to the universal nature of the convenience brought by telemedicine, irrespective of where the patients reside.
The findings of this study have implications for policies and providers in projecting the demand of future use. It is reasonable to wonder whether the current surge of telemedicine is temporary and will vanish after the pandemic.53 Our results suggest that an increase in usage after COVID-19, compared with the volume before COVID-19, is inevitable, confirming the reflection that telemedicine may become a new norm.1
From our sample, 16–21% of the respondents intend to continue using virtual visits after COVID-19 is over, despite being first timers. There is also the possibility that patients will elect to use the telemedicine services more frequently, considering the convenience, particularly reduced wait times. Future longitudinal studies will be necessary to assess patient behavior. Our results from a national sample provide an estimate of the patient volume when life returns to normal. Furthermore, with the increasing trend of internet usage volume, patients would become more comfortable using virtual visits.
Health care practices should allocate a certain level of their resources permanently to handling these continuing demands. This resource may include technology support, the technology that enables two-way data sharing between providers and patients, and the capability to integrate data generated during telemedicine into electronic health records.
In light of the increasing demand and the positive patient and process outcomes, payers may also consider whether to continue covering telemedicine, although this decision will need to be supported by more evidence of clinical outcomes. Despite the overall increasing demand, however, providers of eye-related problems may need to investigate factors contributing to the relatively low satisfaction and provide a better approach for telemedicine services.
The primary limitation of this study is the use of a convenient MTurk sample. Studies have shown that the MTurk population is not representative of the general population with regard to health status and behaviors.54 Compared with a study that used national claims data for acute, nonurgent virtual visit users, our sample is younger (our mean age = 36.4 as opposed to 40.1) and has fewer females (our proportion = 38% as opposed to 61%).32 However, considering that females have higher future use intention than males, our finding of 16–21% of “strongly agree” proportion to use virtual visit in the future may underestimate the true proportion. Despite this limitation of generalizability, our sample is national, and is not limited to a single patient group or a single health care institution.
Another limitation of our study was the subjective measure of waiting time. Since the respondents were eligible as long as they had a virtual visit since January, which spanned 6 months by the time our data collection ended, they may not have crisp memory. However, given our sample size, the mean and median we obtained should still well describe the actual waiting time.
Finally, due to the fact that our questionnaire was designed to get an overall evaluation for all virtual visits for acute conditions during the pandemic, which may involve meetings with multiple physicians with different relationships, pre-existing physician/patient relationship was not measured and examined. Based on literature, we believe that the existing relationship may present a double-edged sword in its influence on satisfaction and future use intention. On the one hand, it may increase satisfaction and intention due to a higher level of trust.55,56 On the other hand, it may make it harder to satisfy the patients due to higher expectation, as predicted by the expectation confirmation theory.57 Future studies may quantify the impact of pre-existing physician/patient relationship on satisfaction and intention.
Conclusion
Eighty-one and a half percent of surveyed patients used virtual visits for the first time during COVID-19. Respondents reported a median waiting time of 2 days for their appointments, and 15 min for the providers after being connected online. Sixteen to 21% of the respondents reported a strong intention to continue using virtual visits after the pandemic is over, with 72–77% reporting positive intention, irrespective of whether they were first timers.
Our results suggest that given the satisfactory process and patient outcomes, the demand for telemedicine during COVID-19 may remain high after the pandemic. However, the supply will be determined based on the evidence of clinical outcomes. This study contributes to the health informatics literature by providing a timely understanding of how patients experience and perceive telemedicine facing acute health conditions using a national sample. To the best of our knowledge, this study is the first to attempt reporting patient waiting time for and during virtual visits, which is an important metric for process outcome.
Authors’ Contributions
Y.C.K.L. was responsible for the study concept and design, survey design, data collection, analysis, interpretation of the data, and initial draft of the article. S.T.T. was responsible for the study concept and design, interpretation of the data, and critically revising the survey and article.
Disclosure Statement
No competing financial interests exist.
Funding Information
This research was, in part, supported by a summer research grant from the Frank G. Zarb School of Business, Hofstra University.
References
- 1. COVID-19 transforms health care through telemedicine: Evidence from the field. J Am Med Inform Assoc 2020;27:1132–1135. Crossref, Medline, Google Scholar
- 2. Clinical-and cost-effectiveness of telemedicine in type 2 diabetes mellitus: A systematic review and meta-analysis. Medicine (Baltimore) 2014;93:e312. Crossref, Medline, Google Scholar
- 3. Telemedicine consultation for patients with diabetes mellitus: A cluster randomised controlled trial. J Telemed Telecare 2018;24:385–391. Crossref, Medline, Google Scholar
- 4. Home telehealth video conferencing: Perceptions and performance. JMIR Mhealth Uhealth 2015;3:e90. Crossref, Medline, Google Scholar
- 5. . The clinical use of Skype—for which patients, with which problems and in which settings? A snapshot review of the literature. Int J Med Inf 2015;84:737–742. Crossref, Medline, Google Scholar
- 6. Telepsychiatry in the assessment and treatment of schizophrenia. Clin Schizophr Relat Psychoses 2014;8:21–27A. Crossref, Medline, Google Scholar
- 7. The potential of technology-based psychological interventions for anorexia and bulimia nervosa: A systematic review and recommendations for future research. J Med Internet Res 2015;17:e85. Crossref, Medline, Google Scholar
- 8. Patient perceptions of video visits using veterans affairs telehealth tablets: Survey study. J Med Internet Res 2020;22:e15682. Crossref, Medline, Google Scholar
- 9. The application of telemedicine to pediatric obesity: Lessons from the past decade. Telemed J E Health 2020 [ahead of print]; DOI: 10.1089/tmj.2019.0314. Google Scholar
- 10. Patient perceptions of telehealth primary care video visits. Ann Fam Med 2017;15:225–229. Crossref, Medline, Google Scholar
- 11. . Using videoconferencing to provide psychological services to rural children and adolescents: A review and case example. J Clin Child Adolesc Psychol 2014;43:115–127. Crossref, Medline, Google Scholar
- 12. . Therapeutic alliance in videoconferencing psychotherapy: A review. Aust J Rural Health 2014;22:280–299. Crossref, Medline, Google Scholar
- 13. Telemedicine evaluations for low-acuity patients presenting to the emergency department: Implications for safety and patient satisfaction. Telemed J E Health 2020;26:1010–1015. Link, Google Scholar
- 14. Efficacy of home-based telerehabilitation vs in-clinic therapy for adults after stroke: A randomized clinical trial. JAMA Neurol 2019;76:1079–1087. Crossref, Google Scholar
- 15. . In-person health care as option B. N Engl J Med 2018;378:104–106. Crossref, Medline, Google Scholar
- 16.
American Hospital Association . The promise of telehealth for hospitals, health systems and their communities, Trend Watch. Washington: American Hospital Association, 2015:46–53. Google Scholar - 17. Interpreting covid-19 and virtual care trends: Cohort study. JMIR Public Health Surveill 2020;6:e18811. Crossref, Medline, Google Scholar
- 18. Rapid deployment of inpatient telemedicine in response to Covid-19 across three health systems. J Am Med Inform Assoc 2020;27:1102–1109. Crossref, Medline, Google Scholar
- 19. Population-level interest and telehealth capacity of US hospitals in response to Covid-19: Cross-sectional analysis of Google search and national hospital survey data. JMIR Public Health Surveill 2020;6:e18961. Crossref, Medline, Google Scholar
- 20. . Virtually perfect? Telemedicine for COVID-19. N Engl J Med 2020;382:1679–1681. Crossref, Medline, Google Scholar
- 21. . Telemedicine in the time of Coronavirus. J Pain Symptom Manage 2020;60:e12–e14. Crossref, Medline, Google Scholar
- 22. , 2020 Available at: https://www.cms.gov/newsroom/fact-sheets/medicare-telemedicine-health-care-provider-fact-sheet (last accessed
May 2, 2020 ). Google Scholar - 23. , 2020. Available at: https://www.hhs.gov/hipaa/for-professionals/special-topics/emergency-preparedness/notification-enforcement-discretion-telehealth/index.html (last accessed
May 2, 2020 ). Google Scholar - 24. . State of telehealth. N Engl J Med 2016;375:154–161. Crossref, Medline, Google Scholar
- 25. Internet videoconferencing for patient–clinician consultations in long-term conditions: A review of reviews and applications in line with guidelines and recommendations. Digit Health 2019;5:1–27. Google Scholar
- 26. . The Coronavirus disease 2019 crisis as catalyst for telemedicine for chronic neurological disorders. JAMA Neurol 2020;77:927–928. Crossref, Medline, Google Scholar
- 27. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc 2020;27:957–962. Crossref, Medline, Google Scholar
- 28. Electronic visits for common acute conditions: Evaluation of a recently established program. Health Aff (Millwood) 2018;37:2024–2030. Crossref, Medline, Google Scholar
- 29. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil 2016;8:3–10. Crossref, Medline, Google Scholar
- 30. Measuring satisfaction and usability of FaceTime for virtual visits in patients with uncontrolled diabetes. Telemed J E Health 2016;22:138–143. Link, Google Scholar
- 31. Patients’ satisfaction with and preference for telehealth visits. J Gen Intern Med 2016;31:269–275. Crossref, Medline, Google Scholar
- 32. . Virtual visits for acute, nonurgent care: A claims analysis of episode-level utilization. J Med Internet Res 2017;19:e35. Crossref, Medline, Google Scholar
- 33. Understanding older people’s readiness for receiving telehealth: Mixed-method study. J Med Internet Res 2018;20:e123. Crossref, Medline, Google Scholar
- 34. . Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Comput Human Behav 2017;77:184–197. Crossref, Google Scholar
- 35. . Comparing Amazon’s Mechanical Turk platform to conventional data collection methods in the health and medical research literature. J Gen Intern Med 2018;33:533–538. Crossref, Medline, Google Scholar
- 36. . Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav Res Methods 2016;48:400–407. Crossref, Medline, Google Scholar
- 37. . Characteristics of personal health information management groups: Findings from an online survey using Amazon’s mTurk. J Med Libr Assoc 2017;105:361–375. Crossref, Medline, Google Scholar
- 38. Amazon’s Mechanical Turk as a viable source for organizational and occupational health research. Occup Health Sci 2018;2:83–98. Crossref, Google Scholar
- 39. . Using Mechanical Turk to study clinical populations. Clin Psycholog Sci 2013;1:213–220. Crossref, Google Scholar
- 40. Using Mechanical Turk to study family processes and youth mental health: A test of feasibility. J Child Family Stud 2015;24:3235–3246. Crossref, Google Scholar
- 41. . Reliability and validity of data obtained from alcohol, cannabis, and gambling populations on Amazon’s Mechanical Turk. Psychol Addict Behav 2017;31:85. Crossref, Medline, Google Scholar
- 42. . Screening Amazon’s Mechanical Turk for adults with ADHD. J Atten Disord 2019;23:1178–1187. Crossref, Medline, Google Scholar
- 43. Consumers’ patient portal preferences and health literacy: A survey using crowdsourcing. JMIR Res Protoc 2016;5:e104. Crossref, Medline, Google Scholar
- 44. What do our patients truly want? Conjoint analysis of an aesthetic plastic surgery practice using internet crowdsourcing. Aesthet Surg J 2017;37:105–118. Crossref, Medline, Google Scholar
- 45. Crowd-sourced assessment of technical skills: An opportunity for improvement in the assessment of laparoscopic surgical skills. Am J Surg 2016;211:398–404. Crossref, Medline, Google Scholar
- 46. Peer Reviewed: Diabetes topics associated with engagement on twitter. Prev Chronic Dis 2015;12:E62. Crossref, Medline, Google Scholar
- 47. . AMIA Annual symposium proceedings. American Medical Informatics Association, 2013. Google Scholar
- 48. . Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav Res Methods 2014;46:1023–1031. Crossref, Medline, Google Scholar
- 49. Detecting and deterring insufficient effort responding to surveys. J Bus Psychol 2012;27:99–114. Crossref, Google Scholar
- 50. A primer on partial least squares structural equation modeling (PLS-SEM), 2nd ed. LA, USA: Sage publications, 2017. Google Scholar
- 51. Computer-using patients want Internet services from family physicians. J Fam Pract 2002;51:570–572. Medline, Google Scholar
- 52. . Willing to wait?: The influence of patient wait time on satisfaction with primary care. BMC Health Serv Res 2007;7:31. Crossref, Medline, Google Scholar
- 53. Telehealth: Simply a pandemic response or here to stay? J Am Med Inform Assoc 2020;27:989–990. Crossref, Medline, Google Scholar
- 54. . Are Mechanical Turk worker samples representative of health status and health behaviors in the US? PloS One 2018;13:e0198835. Crossref, Medline, Google Scholar
- 55. . Impact of the doctor-patient relationship. Prim Care Companion CNS Disord 2015;17: Google Scholar
- 56. Healthcare at your fingertips: The acceptance and adoption of mobile medical treatment services among chinese users. Int J Environ Res Public Health 2020;17:6895. Crossref, Google Scholar
- 57. A cognitive model of the antecedents and consequences of satisfaction decisions. J Market Res 1980:460–469. Crossref, Google Scholar
* https://www.medicaid.gov/medicaid/benefits/telemedicine/index.html
‡ The survey also includes other items, which are for another study and are not reported here.