Patient Experience and Satisfaction with Telemedicine During Coronavirus Disease 2019: A Multi-Institution Experience
Introduction
As the coronavirus disease 2019 (COVID-19) spread across the United States in March 2020, hospitals dramatically reduced in-person patient visits1 and transitioned to telemedicine encounters.2–5 Although virtual visits preceded COVID-19, they typically comprised the minority of all visits and were not studied as the principal form of patient–provider interaction.6,7 The duration of COVID-19 restrictions and the breadth of its impact provided unprecedented opportunities to assess patient satisfaction with telemedicine as the primary modality of patient–physician communication. Recently published studies have assessed telemedicine patient satisfaction in neurosurgical,8–11 orthopedic,2,12 urological,13 and primary care settings.14 However, these studies had <2,000 video visits, measured encounters from early in the COVID-19 pandemic, and focused on one or two departments. In addition, many did not capture or examine ancillary visit characteristics.
Our data set, which includes patient responses from >60,000 unique video visits across two institutions over 19+ weeks, has allowed us to determine the correlation between patient experience and the device used during the encounter, individual wait time, connection interruption, and other factors. To our knowledge, this is the most comprehensive assessment of patient satisfaction with telemedicine to date.
Methods
Telemedicine visit data were gathered from the Hospital for Special Surgery (HSS; NY), adapted from Press-Ganey questionnaires, or from Stanford Health Care (Stanford, CA), from a novel survey unique to telemedicine visits. Data from HSS were aggregated by department and stratified by week. There were no individual-level data. Conversely, Stanford had individual responses. During COVID-19 era, patients at Stanford were sent an electronic survey at the conclusion of their video visit. The complete survey consisted of 12 different patient satisfaction domains (Supplementary Table S1). All respondents received questions on overall visit assessment and their likelihood to recommend (LTR) scores. The remaining 10 sections were evenly randomized such that each respondent only saw the questions of two sections. Randomization ensured that an even number of respondents would see each question section. Each response matched to a unique video visit. No demographic data were captured. The percentage of respondents who reported the highest possible LTR score (“LTR top box percentage”) were tracked longitudinally to measure differences in patient satisfaction over time. All Press-Ganey Hospital Consumer Assessment of Healthcare Providers and Systems variables are reported using the LTR top box percentage score as a key metric of patient satisfaction, along with overall visit score, and, therefore, this outcome measure was selected to be included in the present study.
Univariable and multivariable ordinal logistic regression analyses were conducted to identify negative and positive predictors of overall visit scores. Because patients were randomly assigned certain questions rather than the complete survey, the number of observations for a given variable is a percentage of the total 60,550 visits (see Table 1 for the number of observations for a given variable). Therefore, for each variable, an “unknown” category was encoded to allow multivariable comparison in a full model. The multivariable model was constructed in a forward stepwise manner to obtain a more parsimonious model. All analyses were conducted in R version 3.5.3 (The R Foundation) or STATA (StataCorp, USA). Significance was set at 0.05, and all tests were two sided. As the data were deidentified, the study was deemed exempt by both institutions’ institutional review boards.
VARIABLES | NUMBER |
---|---|
N | 60,550 |
LTR [mean (SD)] | 4.61 (0.73) |
LTR top box percentage [mean (SD)] | 0.72 (0.45) |
Device used, n (%) | |
Cell phone | 33,325 (55.0) |
Computer | 17,488 (28.9) |
Tablet | 7,147 (11.8) |
Unknown | 2,590 (4.3) |
Lost connection, n (%) | 11,191 |
Did not happen | 10,096 (90.2) |
I lost the connection | 1,095 (9.8) |
First telemedicine encounter?, n (%) | 11,234 |
No | 7,242 (64.5) |
Yes | 3,992 (35.5) |
Wait time, n (%) | 6,983 |
0–5 min | 4,874 (69.8) |
6–20 min | 1,730 (24.8) |
> 20 min | 379 (5.4) |
Received phone reminder?, n (%) | 11,046 |
No | 3,962 (35.9) |
Yes | 7,084 (64.1) |
Help needed to connect?, n (%) | 11,352 |
No | 10,326 (91.0) |
Yes | 1,026 (9.0) |
Moved to regular phone call?, n (%) | 11,197 |
No | 9,961 (89.0) |
Yes | 1,236 (11.0) |
Connected before the visit was scheduled to begin?, n (%) | 11,393 |
No | 798 (7.0) |
Yes | 10,595 (93.0) |
Results
Between June 22, 2020, and November 1, 2020, there were 60,550 telemedicine visits linked with a postvisit survey across 93 services at Stanford. At HSS, there were 66,349 total telemedicine visits with 7,348 linked with a postvisit survey. At Stanford, the weekly average was 3,187 visits (range: 2,557–3,462). At HSS, the weekly average was 210 visits (range: 124–414). Over 19 weeks, the mean overall visit score for telemedicine encounters at Stanford increased from 4.69/5.00 (standard deviation [SD]: 0.67) to 4.77/5.00 (SD: 0.54), a gain of 1.7% (p < 0.0001) (Fig. 1A), and the LTR top box percentage increased from 69.6% to 74.0%, a gain of 6.3% (p = 0.0002) (Fig. 1B). The LTR top box percentage at HSS showed no significant change across 35 visit weeks (p = 0.7067) (Fig. 1C). At Stanford, cellular telephones were the most common method of communicating during the telemedicine encounter, comprising 33,325 (55.0%) of all video visits. Computers (n = 17,488; 28.9%) and tablets (n = 7,147; 11.8%) were used less frequently (Table 1).
The remaining question sections were randomly distributed to all patients at Stanford. Of the 11,191 patients who received and answered questions relating to their visit’s internet connection, 1,095 (9.8%) reported that the connection was lost (Table 1). Of the 11,234 patients who received and answered whether this telemedicine encounter was their first video visit, 3,992 (35.5%) answered affirmatively. We observed that the majority of surveyed patients experienced wait times <5 min (69.8%). However, 9.0% (n = 1,026) required help to connect and 11.0% (n = 1,236) were forced to move the telemedicine encounter to a regular phone call. The number of telemedicine encounters listed by selected departments at Stanford is given in Supplementary Table S2.
Week-by-Week LTR Top Box Percentages
The percentage of respondents who listed the highest LTR option for each week was graphed and stratified by department. Linear trend lines were interpolated to determine whether the LTR score trend significantly increased or decreased over time. Supplementary Figure S1A–L lists weekly LTR top box percentages for several nonsurgical departments at Stanford. Despite some weekly fluctuation, LTR trends remained relatively consistent across specialties and time, with no trend line deviating significantly from 0, with the exception of neurology, whose LTR trend increased (p = 0.0121). LTR trend stability was recapitulated across four surgical specialties (Supplementary Fig. S2) and five oncological services (Supplementary Fig. S3).
Univariable and Multivariable Analyses
At the univariable level, multiple factors were significantly associated with overall visit score (Table 2). The use of a cell phone (odds ratio [OR]: 1.18; 95% confidence interval [CI]: 1.13–1.23) or tablet (OR: 1.19; 95% CI 1.12–1.28) was associated with mildly increased visit score (p < 0.0001), as was the receipt of a phone reminder (OR: 1.24; 95% CI: 1.13–1.36) (p < 0.0001). Predictably, interrupted connection (OR: 0.31; 95% CI: 0.27–0.35), needed connection assistance (OR: 0.42; 95% CI: 0.37–0.48), and increased wait time (>20 min) (OR: 0.32; 95% CI: 0.26–0.40) reduced overall visit scores (p < 0.0001). Those patients who had to switch to a regular phone call for their telemedicine encounter also reported lower overall scores (OR: 0.28; 95% CI: 0.24–0.31) (p < 0.0001). In similarly expected results, patients who indicated they were less able to see or understand the provider, or who indicated the provider was not able to properly see, understand, or examine them, were associated with lower overall scores.
CHARACTERISTIC | UNIVARIABLE | |||
---|---|---|---|---|
OR | 95% CI | p | N | |
Number of weeks | 1.01 | 1.01–1.01 | <0.0001 | 59,103 |
First video visit | ||||
No | 0.00 | — | — | 7,242 |
Yes | 0.91 | 0.83–1.00 | 0.062 | 3,992 |
Device used | ||||
Computer | 0.00 | — | — | 17,488 |
Cell phone | 1.18 | 1.13–1.23 | <0.0001 | 33,325 |
Tablet | 1.19 | 1.12–1.28 | <0.0001 | 7,147 |
Interrupted connection | ||||
No | 0.00 | — | — | 10,096 |
Yes | 0.31 | 0.27–0.35 | <0.0001 | 1,095 |
Phone reminder | ||||
No | 0.00 | — | — | 3,962 |
Yes | 1.24 | 1.13–1.36 | <0.0001 | 7,084 |
Wait time | ||||
0–5 min | 0.00 | — | — | 4,874 |
6–20 min | 0.56 | 0.50–0.64 | <0.0001 | 1,730 |
>20 min | 0.32 | 0.26–0.40 | <0.0001 | 379 |
Help needed to connect | ||||
No | 0.00 | — | — | 10,326 |
Yes | 0.42 | 0.37–0.48 | <0.0001 | 1,026 |
Scheduling difficulty | ||||
Not difficult at all | 0.00 | — | — | 9,687 |
Slightly difficult | 0.44 | 0.38–0.52 | <0.0001 | 853 |
Moderately difficult | 0.34 | 0.28–0.42 | <0.0001 | 457 |
Very difficult | 0.19 | 0.14–0.27 | <0.0001 | 115 |
Extremely difficult | 0.25 | 0.16–0.37 | <0.0001 | 102 |
Connected before visit | ||||
After visit scheduled to begin | 0.00 | — | — | 798 |
Before visit scheduled to begin | 1.37 | 1.16–1.61 | <0.0001 | 10,595 |
Move to regular phone call | ||||
No | 0.00 | — | — | 9,961 |
Yes | 0.28 | 0.24–0.31 | <0.0001 | 1,236 |
How well were you able to see the provider? | ||||
Extremely well | 0.00 | — | — | 7,796 |
Very well | 0.19 | 0.17–0.22 | <0.0001 | 2,738 |
Moderately well | 0.09 | 0.08–0.11 | <0.0001 | 448 |
Slightly well | 0.04 | 0.03–0.07 | <0.0001 | 86 |
Not well at all | 0.05 | 0.04–0.07 | <0.0001 | 305 |
How well did the provider see you? | ||||
Extremely well | 0.00 | — | — | 5,563 |
Very well | 0.21 | 0.19–0.24 | <0.0001 | 4,278 |
Moderately well | 0.09 | 0.07–0.10 | <0.0001 | 948 |
Slightly well | 0.04 | 0.03–0.05 | <0.0001 | 133 |
Not well at all | 0.05 | 0.04–0.06 | <0.0001 | 339 |
How well did the provider see the indicated area? | ||||
Extremely well | 0.00 | — | — | 1,093 |
Very well | 0.27 | 0.21–0.36 | <0.0001 | 1,133 |
Moderately well | 0.13 | 0.09–0.17 | <0.0001 | 460 |
Slightly well | 0.05 | 0.04–0.08 | <0.0001 | 103 |
Not well at all | 0.05 | 0.03–0.08 | <0.0001 | 96 |
How well did the provider understand you? | ||||
Extremely well | 0.00 | — | — | 8,227 |
Very well | 0.15 | 0.13–0.16 | <0.0001 | 2,743 |
Moderately well | 0.03 | 0.02–0.04 | <0.0001 | 247 |
Slightly well | 0.01 | 0.01–0.02 | <0.0001 | 59 |
Not well at all | 0.00 | 0.00–0.01 | <0.0001 | 94 |
How well did you understand the provider? | ||||
Extremely well | 0.00 | — | — | 8,088 |
Very well | 0.14 | 0.12–0.15 | <0.0001 | 2,804 |
Moderately well | 0.03 | 0.03–0.04 | <0.0001 | 307 |
Slightly well | 0.01 | 0.01–0.02 | <0.0001 | 60 |
Not well at all | 0.00 | 0.00–0.01 | <0.0001 | 92 |
After adjusting for a number of covariates, including the number of weeks, device used, and factors related to visit connectivity, the associations that were observed at the univariable level were recapitulated in a multivariable model (Fig. 2). The use of a cell phone (adjusted odds ratio [aOR]: 1.18; 95% CI: 1.12–1.23) and tablet (aOR: 1.15; 95% CI: 1.07–1.23) was again associated with higher overall score (p < 0.0001), and wait times beyond 5 min were linked with markedly reduced scores. Interestingly, first-time telemedicine users were not linked to a reduction in the odds of expressing a perfect (“5/5”) overall score (p = 0.760). Instead, those parameters related to patient difficulty with the logistics of the telemedicine encounter (help needed or scheduling difficulty) or the integrity of the visit itself (lost connection; moved to regular phone call) were linked to dramatically reduced overall score. In addition, reductions in the patient’s self-reported ability to see the provider and their perception of the physician’s ability to examine them were both associated with substantially lower overall visit scores. In this model, an encounter with an interrupted connection decreased the odds of reporting a perfect overall score by >50% (p < 0.0001), whereas an encounter that was forced to be moved to a regular phone call decreased the odds by 60% (p < 0.0001), holding all other parameters constant. In contrast, patients who connected before the visit was scheduled to begin were 1.79 × more likely to report a perfect overall score (p < 0.0001).
At HSS, top box LTR percentage was significantly correlated with audio connection quality scores (p = 0.002), video connection quality scores (p < 0.001), and ease of talking with the clinical provider over video scores (p = 0.003) (Supplementary Fig. S4). These HSS data accord with the results from Stanford, shown in Figure 2.
Discussion
We present the largest published description of patient satisfaction with telemedicine during the COVID-19 era. In two separate institutions, we found high satisfaction with telemedicine encounters across multiple measures. At Stanford, mean overall visit score and LTR top box percentages increased over the observation period, whereas LTR top box percentages at HSS remained stable over 35 weeks. When stratifying by individual department, LTR top box percentage trends remained fairly consistent over the 4 months of observation and across multiple medical domains, including surgical and nonsurgical specialties. Average LTR top box scores were even similar between inpatient and outpatient services, again suggesting broad satisfaction with telemedicine.
Although previous studies have shown that patient satisfaction with telemedicine visits equals, or betters, that with in-person encounters,15,16 few have looked across institutions, over extended time periods, or examined telemedicine-specific factors. Given the unprecedented public health crisis that continues to exist throughout the United States, we believe our findings are of substantial utility. Using data from >60,000 separate responses, we identified a number of important telemedicine-specific factors that predict increased overall visit score. These include the use of cell phones or tablets, phone reminders, and connecting before the visit is scheduled to begin. Just as meaningful, however, were the negative predictors of overall visit score. Visits with poor connectivity; extended wait times; or difficulty being seen, understood, or examined by the provider were linked with reduced odds of high scores. A patient’s status as a first-time telemedicine user was not significantly associated with overall visit score after controlling for available covariates.
Our results suggest that attention to connectivity, audio/visual definition, and wait time will help optimize patient satisfaction with telemedicine encounters. Increased difficulty seeing and understanding the provider strongly predicted lower overall scores. Visits with interrupted connections and those visits that were moved to regular phone calls were associated with markedly reduced patient satisfaction. Importantly, patient perception of provider difficulty examining or understanding the patient led to dramatically lower overall scores. To achieve high overall scores, physicians may utilize strategies to avoid negative outcomes, including phone reminders. In addition, clinic schedulers can instruct patients to ensure they are able to connect to the telemedicine platform before their scheduled encounter with the physician. From our study, doing so results in a 1.8 × increase in the likelihood of reporting the highest visit score.
Previous patient satisfaction studies have documented that even small increases (1–10%) in patient scores may be clinically relevant.16–18 In fact, the method of capturing patient preferences is highly sensitive to small absolute changes in overall or LTR scores.19 For example, in 2015 for ophthalmology, a reduction in the mean Press-Ganey score from 92.4% to 91.7% corresponded to a decrease from the 70th percentile to the 25th percentile.20 Nationally, in 2012, an absolute 3% decrease in mean composite score translated to a drop in 40 percentiles.19 The magnitude of the associations identified in our multivariable model falls well within the range of purported clinical significance.
In March 2020, the Centers for Medicare and Medicaid Services (CMS) determined that providers would be compensated equally for telemedicine or in-person appointments. The Health Insurance Portability and Accountability Act (HIPAA) regulations on telemedicine encounters were waived and other legal barriers were suspended to facilitate the widespread adoption of virtual medicine. It is uncertain whether these legal determinations will be extended after the COVID-19 era concludes. Regardless, as a larger share of physician–patient interactions continues to be conducted through telemedicine,16,21 it becomes increasingly critical to understand the factors that influence and, ultimately, predict patient satisfaction. These visit characteristics are of interest not only to physicians and hospitals, but also to the CMS and private insurers who will determine how virtual encounters are compensated in the coming years. Our results suggest that certain telemedicine-specific factors have a significant impact on patient satisfaction, including audio/visual connectivity and ease of use. It is hoped that our study identifies key levers of patient satisfaction with virtual encounters and lays the foundation for future focused interventions to improve the telemedicine experience.
Limitations
As with any survey-based study, our analyses are limited by nonresponder bias. Previous research indicates that satisfied patients are more likely to respond to surveys.22 In addition, we were unable to control for patient-specific factors that might influence their responses, including disease diagnosis or severity, gender, race, ZIP code, or income. However, we attempted to mitigate this bias through the random assignment of question sections across thousands of respondents. Second, since our data were derived from two tertiary care centers, our trends might not extend to other hospitals or clinic settings. Third, the LTR top box percentages for all in-person encounters were uniformly higher than those for telemedicine visits (Supplementary Fig. S5). However, we cannot make the claim that patients favored in-person to telemedicine encounters from these data alone; the survey instruments used to assess telemedicine versus in-person encounters were different. In addition, patient-specific factors (age, gender, diagnosis, and disease severity) are nonrandomly distributed among in-person versus telemedicine visits. Unfortunately, the literature contains few rigorous comparisons between telemedicine and in-person encounters to guide this discussion.16 Fourth, the survey used at Stanford to assess patient satisfaction with telemedicine was not a Press-Ganey instrument. Even though some of the questions in Stanford’s survey were phrased in the same manner as Press-Ganey questions, and included the same answer options, their psychometrics are different. However, in this study, we refrained from making absolute comparisons with data from other studies and instead focused on relative changes (positive/negative associations or positive/negative predictors). Finally, although previous studies have identified associations between increased patient satisfaction and treatment adherence,23 surgical readmission,24 and patient retention,25 it remains unclear whether patient satisfaction predicts or results in improved patient care.26
Conclusions
Using patient satisfaction data from two large academic centers, we demonstrated that patients remained broadly satisfied with telemedicine across surgical, nonsurgical, inpatient, and outpatient services during the COVID-19 pandemic. In addition, we identified telemedicine-specific factors that significantly impact patients’ satisfaction with the virtual encounter. To increase patient satisfaction during a telemedicine visit, attention should be paid to video connectivity, audio/visual definition, and ease of use.
Acknowledgments
We thank Joshua Frost and Michael Yeung from the Stanford Health Care Patient Experience Team for assisting in data access and collation.
Disclosure Statement
No competing financial interests exist.
Funding Information
No funding was received for this article.
Supplementary Material
References
- 1. Hospital Volumes of 5 Medical Emergencies in the COVID-19 Pandemic in 2 US Medical Centers. JAMA Intern Med 2021;181:272–274. Crossref, Medline, Google Scholar .
- 2. Telemed J E Health 2020; [Epub ahead of print]; 10.1089/tmj.2020.0255. Google Scholar The use and acceptance of telemedicine in orthopedic surgery during the COVID-19 pandemic.
- 3. Telemedicine: Patient-provider clinical engagement during the COVID-19 pandemic and beyond. J Gastrointest Surg 2020;24:1692–1697. Crossref, Medline, Google Scholar .
- 4. The impact of COVID-19 on telemedicine utilization across multiple service lines in the United States. Healthcare (Basel) 2020;8:380. Crossref, Medline, Google Scholar .
- 5. The rapid increase in telemedicine visits during COVID-19. Patient Exper J 2020;7:72–79. Crossref, Google Scholar .
- 6. Telemedicine: Past, present, and future. Cleve Clin J Med 2018;85:938–942. Crossref, Medline, Google Scholar .
- 7. Assessing telemedicine utilization by using medicaid claims data. Psychiatr Serv 2017;68:173–178. Crossref, Medline, Google Scholar .
- 8. Telemedicine in the era of coronavirus disease 2019 (COVID-19): A neurosurgical perspective. World Neurosurg 2020;139:549–557. Crossref, Medline, Google Scholar
- 9. Telemedicine in neurosurgery: Lessons learned from a systematic review of the literature for the COVID-19 era and beyond. Neurosurgery 2020;88:e1–e12. Crossref, Medline, Google Scholar
- 10. Telemedicine in neurosurgery: lessons learned and transformation of care during the COVID-19 pandemic. World Neurosurg 2020;140:e387–e394. Crossref, Medline, Google Scholar
- 11. Patient satisfaction with neurosurgery telemedicine visits during the coronavirus disease 2019 pandemic: A prospective cohort study. World Neurosurg 2020;145:e184–e191. Crossref, Medline, Google Scholar
- 12. Patient and surgeon satisfaction with telehealth during the COVID-19 pandemic. Bull Hosp Joint Dis (2013) 2020;78:227–235. Medline, Google Scholar
- 13. Preliminary assessment of patient and physician satisfaction with the use of teleconsultation in urology during the COVID-19 pandemic. World J Urol 2020; [Epub ahead of print]; doi:
10.1007/s00345-020-03432-4 Crossref, Google Scholar - 14. Implementation of video visits during COVID-19: Lessons learned from a primary care practice in New York City. Front Public Health 2020;8:514. Crossref, Medline, Google Scholar
- 15. Randomized controlled clinical trial of “virtual house calls” for Parkinson disease. JAMA Neurol 2013;70:565–570. Crossref, Medline, Google Scholar
- 16. Patient satisfaction with telemedicine during the COVID-19 pandemic: Retrospective cohort study. J Med Internet Res 2020;22:e20786. Crossref, Medline, Google Scholar
- 17. Using Press Ganey Provider Feedback to improve patient satisfaction: A pilot randomized controlled trial. Acad Emerg Med 2017;24:1051–1059. Crossref, Medline, Google Scholar
- 18. An observation unit may help improve an Institution’s Press Ganey satisfaction score. Crit Pathw Cardiol 2011;10:104–106. Crossref, Medline, Google Scholar
- 19. The association between Asian patient race/ethnicity and lower satisfaction scores. BMC Health Services Res 2020;20:678. Crossref, Medline, Google Scholar
- 20. Factors associated with patient Press Ganey satisfaction scores for ophthalmology patients. Ophthalmology 2016;123:242–247. Crossref, Medline, Google Scholar .
- 21. Why telemedicine is here to stay. BMJ 2020;371:m3603. Crossref, Medline, Google Scholar .
- 22.
Press Ganey Administration of Hospital Consumer Assessment of Healthcare Providers and Systems Survey Result in a Biased Responder Sample for Hip and Knee Arthroplasties . J Arthroplasty 2019;34:2538–2543. Google Scholar - 23. Impact of patient satisfaction with his or her inhaler on adherence and asthma control. Allergy Asthma Proc 2018;39:437–444. Crossref, Medline, Google Scholar
- 24. Is quality important to our patients? The relationship between surgical outcomes and patient satisfaction. BMJ Qual Safety 2018;27:48–52. Crossref, Medline, Google Scholar
- 25. Switching doctors: Predictors of voluntary disenrollment from a primary physician’s practice. J Fam Pract 2001;50:130–136. Medline, Google Scholar .
- 26. Patient satisfaction and patient-centered care: Necessary but not equal. JAMA 2012;308:139–140. Crossref, Medline, Google Scholar .