Development and Validation of an Instrument in Spanish to Evaluate Patient Satisfaction in Telemedicine Consultation During COVID-19 Pandemic
Introduction
The severe acute respiratory syndrome, coronavirus 2, SARS-CoV-2, is a new virus that causes COVID-19, a disease that mainly affects the respiratory tract. This coronavirus is mainly transmittable from person to person through droplets of respiratory secretions, aerosols, and by direct, indirect, or close contact of human to human.1 In March 2020, the World Health Organization (WHO) declared a pandemic state due to this virus.2 As of January 2021, >100 million cases and >2 million deaths were quantified worldwide.3
In this context, telemedicine was one of the ways of medical care chosen by multiple health institutions to continue providing consultation services, reducing the risk of patients’ exposure by not requiring them to attend the clinic and hospital facilities.4–6 Governments and industries have shown great enthusiasm for telemedicine due to the health outcomes of this type of intervention.
Since April 2020, the Hospital Clinica Nova implemented a telemedicine program with two remote consultation modalities, a telephone consultation system and a video call consultation system using a mobile application. In addition, a drive-thru system to deliver medicine was created to minimize contact between large number of patients.
As part of the evaluation process of the telemedicine program implemented in the hospital, a bibliographic instrument search was carried out to measure the satisfaction of patients treated through telemedicine. However, no validated scale was found in Spanish that would allow obtaining reliable data on this part of the patient-centered domain of quality in health care.
Therefore, the objective of this study was to develop and validate an instrument that allows measuring patient satisfaction with telemedicine consultations and knowing the level of satisfaction with this type of remote consultation that was implemented at the Hospital Clinica Nova during the COVID-19 epidemic.
Materials and Methods
This was a qualitative study for developing and validating a scale to evaluate patient satisfaction with telemedicine consultation by doctors of Hospital Clinica Nova during the COVID-19 pandemic. It was carried out following the Standards for Reporting Qualitative Research.7 This approach includes a qualitative phase that comprehends item development and the first step of scale validation. Also, a quantitative phase was included for the scale development and evaluation.
A team conducted the process made up of quality and patient safety professionals and supported by a research methodology expert. The study was carried out in the COVID-19 pandemic, including patients treated by telemedicine from April to August 2020. The scale development and validation were performed from September to October 2020.
A homogeneous population of patient’s was selected since we divided patients according to teleconsultation modality (video call or telephone consultation) from database in half and then randomly selected them for participation. We contacted the patients through telephone calls to answer the questionnaire. To control memory bias, patients where asked whether they remembered the experience from the last consultation, if they did not remember, they were not invited to answer the questionnaire. This gives internal validity to the study, but with sufficiently general characteristics that allow generalizing the results to an external population, made up of adult patients who have received care through either of the two teleconsultation modalities (video call or telephone consultation) offered by the Hospital Clinica Nova.
The sample size was calculated at a rate of 10 patients per item, following Boateng et al.,8 resulting in a sample of 120 patients.
This study was carried out in full compliance with the ICH E6 guide of Good Clinical Practices and the principles of the Declaration of Helsinki, and applicable Mexican regulations. It was submitted to the local institutional review board (registration number CEI 03-2020).
The instrument development and validation process were an adaptation of the methodology proposed by Boateng et al. in the document Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research,8 and it was divided into three phases: (1) item development, (2) scale development, and (3) scale evaluation (Fig. 1).
Fig. 1. Scale development and validation process.
Phase 1: Item Development
The first phase was qualitative and was divided into two steps: identifying the study domain and the generation of the scale items. The domain of the study was patient satisfaction with telemedicine consultations. Once defined, it was confirmed that there were no other scales that would fulfill the same purpose. Based on a literature review, scales, and existing indicators, the deductive method was used to create the first items.
The second step consisted of content validation; this was carried out by two different methods: the first through submission to validation by expert judges using the Delphi method and the second through interviews with patients to achieve apparent validation. For the content validity, a group of experts was selected, including doctors who provide consultations through telemedicine, experts in health care quality, in organizational development, in research methodology, in health administration, in bioethics, in market psychology, in perspective, in health technologies, and telemedicine. Two rounds of the Delphi study were conducted to reach a consensus >50%.
The second method used to obtain content validation was to evaluate the items’ apparent validity, for which telephone interviews were conducted with the target population. The objective of the apparent validation was to improve the drafting of the items to facilitate their understanding, to identify confusing questions, and to allow the participants to improve their clarity, recognize difficult questions, corroborate that the answer options were adequate, and determine whether there was a problem with the order of the items.
At the end of this first phase, a 12-item questionnaire was obtained, 11 on a 7-point Likert scale as they were bipolar questions, that is, they reflect the grade of two different qualities (for example, “completely satisfied” and “completely dissatisfied).” Likewise, a question was included on a Net Promoter Score (NPS) scale from 1 to 10, a scale designed to evaluate customer experience and predict acceptance and business growth.
Phase 2: Scale Development
For the next phase, the scale was applied by telephone using the sample size proposed by Boateng et al.8 at a rate of 10 patients per item, resulting in a sample of 120 patients. Once the results were obtained and interitem and item-total correlation analyses were performed as item reduction tests, those items with a correlation ≤0.22 were eliminated.
Subsequently, the stability validation was carried out through repeatability and reproducibility tests (test–retest). Intraoperator stability (repeatability) was analyzed through the consecutive application of the scale, 24 h apart, to the same patient by the same evaluator. The same scale was applied to the same person with a different evaluator after 24 h. To analyze interoperator stability (reproducibility), Spearman’s correlation test was used for this analysis.
Phase 3: Evaluation of The Scale
The last phase consisted of evaluating the scale; to do this, two tests were used: the first test was a reliability assessment by a variability and standard deviation (SD) study using Cronbach’s alpha. An alpha coefficient of 0.70 was considered adequate. The second test was used to determine the number of factors or dimensions of the scale, and a factorial analysis was carried out with the Kaiser Meyer Olkin (KMO) tests (Kaiser, Meyer, and Olkin), Bartlett’s test of sphericity, and varimax rotation.
Data Analysis
The patient’s information was anonymized. Exploration of normality with the Shapiro–Wilk test was conducted, and exponential transformation was performed when appropriate. Descriptive statistics were expressed in frequencies, percentages, means, and SD for categorical and quantitative variables. For construction and validation of the measurement instrument, the following statistical tests were carried out: For item reduction, and interitem and item-total correlation analyses were performed; a Spearman correlation test for intraoperator performed stability validation (test–retest), and a Kristen Wallis test for interoperator; to analyze the internal reliability of the survey, the Cronbach’s alpha test was used; finally, factor extraction was carried out through factor analysis. A value of p ≤ 0.05 will be considered statistically significant. Missing values that were completely random were eliminated from the analyses. SPSS IMB 25 and Microsoft Excel 16.42 programs were used for data analysis
Results
Based on the literature review and through the deductive method, a first questionnaire was constructed with 10 items, 9 on a Likert scale from 1 to 7 (1 no satisfaction, 7 high satisfaction) and 1 on an NPS scale from 1 to 10 that included different components regarding patient satisfaction. The questionnaire was sent to 15 expert judges for content validation using the Delphi methodology. After two consensus rounds, one of the originally posed questions was removed, and the experts added three new proposals. The apparent validation through patient interviews allowed the adjustment of three questions that contained complex words and had confusing syntax. At the end of the first phase (development of the items), a 12-item questionnaire was obtained, as given in Table 1.
| CONSECUTIVE NUMBER | ITEM |
|---|---|
| 1 | ¿Qué tan fácil fue para usted solicitar la consulta telefónica/de telemedicina? Siendo 7 Muy Difícil y 7 Muy Fácil. |
| 2 | ¿Que tan satisfecho está con el tiempo que pasó desde que solicitó la consulta hasta que fue atendido? Siendo 1 Nada Satisfecho y 7 Muy Satisfecho. |
| 3 | ¿Qué tan cómodo se sintió durante la consulta telefónica/de telemedicina? Siendo 1 Nada Cómodo y 7 Muy Cómodo. |
| 4 | ¿Qué tanto pareció preocuparse el médico por usted como persona? Siendo 1 Muy Poco y 7 Mucho. |
| 5 | ¿Qué tan bien funcionó la aplicación de telemedicina/el equipo telefónico durante la consulta? Siendo 1 Muy Mal y 7 Muy Bien. |
| 6 | ¿Considera que la consulta telefónica/de telemedicina le ayudó a mejorar su estado de salud? Siendo 1 Para Nada y 7 Mucho. |
| 7 | ¿Considera confiable la forma en que el médico diagnostica y trata a los pacientes a través de consulta telefónica/telemedicina? Siendo 1 Nada Confiable y 7 Muy Confiable. |
| 8 | ¿Considera que la consulta telefónica/por telemedicina fue tan buena como una consulta regular en persona? Siendo 1 Nada Buena y 7 Igual de Buena. |
| 9 | ¿Qué tan claro le quedó lo que el médico le dijo sobre su padecimiento y lo que debería hacer para tratarlo? Siendo 1 Nada Claro y 7 Muy Claro. |
| 10 | Si tuviera la necesidad de consultar al médico en otra ocasión ¿Le gustaría volver a consultar por teléfono/telemedicina? Siendo 1 Para Nada y 7 Mucho. |
| 11 | ¿En general, qué tan satisfecho estuvo con la última consulta telefónica/de telemedicina que recibió? Siendo 1 Nada Satisfecho y 7 Muy Satisfecho. |
| 12 | Considerando su experiencia completa ¿Qué tan probable sería que recomendara el servicio de telemedicina a un amigo o familiar? Siendo 1 Nada Probable y 10 Muy Probable. |
For the scale development phase, the research team contacted 152 patients; we excluded 32 patients in relation that they were not interested in participating or did not remember their experience in video call or telephone consultation. The final sample was 120 patients, of which 74 were women (67%) and 46 men (33%). The age range was from 19 to 94 years, with a mean of 52.7 years and a SD of 20.09. There were 78 different reasons for consultation in relation to internal medicine pathologies. The most frequent were hypertension (n = 14), type 2 diabetes (n = 11), the review of laboratory or radiology examinations (n = 6), depression (n = 4), and hypothyroidism (n = 6). The application time ranged from 3 to 12 min, depending on the patient’s responses. In the distribution according to specialty, internal medicine accounted for 75% of the responses. The rest of the specialties made up the other 25%. According to the type of consultation, the surveyed patients were distributed as follows: 52% were first-time patients and 48% were subsequent. The type of appointment was as follows: 45% video meetings and 55% telephone consultation.
Item reduction: interitem and item-total correlation tests were performed to eliminate those items that presented a low correlation (≤0.22). At the end of this test, the result was the elimination of three items, leaving a total of nine. The mean of the correlation between elements (item-total) of the remaining nine items was 0.456.
Stability tests: the test–retest results between the same evaluator showed a significant correlation in six of the nine questions. Similarly, in the test–retest between different evaluators, a significant correlation was obtained in six of the nine items. Subsequently, the intraclass correlation test between the first application, 0.858, and the second, 0.938, was analyzed, which indicates that both applications show high reliability regardless of the internal correlation shown in the previous test. A Kruskal–Wallis test was also performed for independent samples to identify differences between evaluators, finding that there was no significant difference between them.
Scale evaluation: in the reliability test, a Cronbach’s alpha of 0.86 was obtained. For the extraction of factors from the scale, a factorial analysis was performed in which Bartlett’s test of sphericity <0.05 and a KMO measure of 0.842 were obtained. However, when using the component analysis method with varimax rotation, two factors were identified within the questionnaire (Fig. 2), one factor was integrated of items 3, 4, 9, and 11, four items related to the patient’s perception of the service, the other factor was comprehended of items 6, 7, 8, 10, and 12, these five were more related with results of the consultation.
Fig. 2. Item distribution by factors. Factor 1 was integrated by four items related with patients’ perception such as comfort and general satisfaction with the telemedicine consultation. Factor 2 comprehended five items related with results appreciation such as capacity of telemedicine consultation for improving patient’s health.
Evaluation Of Patient Satisfaction
The 8 items on a Likert scale from 1 to 7 had a mean (SD) of 6.35 (1.01). The item on the NPS scale from 1 to 10 had a mean of 8.55 (1.88), as given in Table 2.
Responses between men and women did not have significant differences, except for item number 4, where women showed a higher result than men, mean of 6.72 versus 6.41, respectively (p ≤ 0.05). Higher scores were found in items 4, 7, and 11 for patients seen in subsequent consultations, compared with the first time (mean of 6.72 vs. 6.48, 6.44 vs. 6.15, 6.71 vs. 6.40, respectively p ≤ 0.05). In the same items, the patients that attended telephone consultation had higher results than those attended by video-meeting (item 4, mean of 6.74 vs. 6.43; item 7, 6.46 vs. 6.07; item 11, 6.71 vs. 6.35, p ≤ 0.05).
Discussion
This study developed and validated a highly reliable Spanish instrument that assesses patient satisfaction with telemedicine consultations; this instrument may be applicable in other Spanish-speaking counties. The study showed that in Hospital Clinica Nova, satisfaction with telemedicine medical care was high, so this type of consultation can be a valid strategy to attend to patients in a pandemic context.
Some strategies that were taken to prevent the accelerated transmission of SARS-CoV-2 among the population were the cancellation of nonessential activities and the restriction of places where crowds would gather around. In Mexico, as in other parts of the world, multiple health institutions and hospitals prioritized the care of patients with COVID-19.9 Likewise, they had to rethink how they would continue providing outpatient services without conditioning a high risk of patient exposure to the virus.
Telemedicine was an answer for health institutions that enabled them to continue providing consultation services of many medical specialties while reducing the risk of exposure of patients and families by not requiring them to attend the facilities unless it was quite necessary.
Governments and industries have shown great enthusiasm for telemedicine due to the health outcomes of this type of intervention. Although more extensive studies are needed on the subject, it has been shown that, at least in the short term, telemedicine improves the clinical outcomes of patients with chronic diseases.10
There is a background of telemedicine in Mexico since the 1970s through rural clinics where medical advice was given through radio links.11 Since then, telemedicine has evolved in the country; however, the focus has always been to improve access to health services in remote rural and indigenous areas, where there is insufficient coverage.
Satisfaction has been a critical indicator often used as part of the quality assessment of health care.12 For some authors, quality in service industries is directly related to user satisfaction, which is defined as an evaluation process where consumers compare their expectations against their perceptions.13 One of the main examples of this way of evaluating quality is the SERVQUAL model.
López et al. define quality in health care as one that allows the patient to receive a correct diagnosis and treatment that will lead to the optimal health status achievable according to current knowledge of medical science and the biological factors of the patient. Using low cost strategies, the patient should be exposed to the minimum harm, and the maximum satisfaction.14
Although it is well known that the quality of clinical care cannot be evaluated only by knowing patient satisfaction, this measurement is an important element of the patient-centered domain of health care quality15 and allows health providers to identify opportunities for improvement, a better understanding of the patients, as well as their needs.
Likewise, patient satisfaction has been related to better adherence to treatments, better prognoses and clinical results, less use of health services, and less litigation for malpractice. However, it is often difficult to find studies with sufficient scientific rigor to evaluate patient satisfaction.16
After following the methodology for the elaboration and validation of a documentary measurement instrument, it was possible to obtain a valid and reliable scale of patient satisfaction regarding the telemedicine consultations for the population of the Hospital Clinica Nova.
In general, good satisfaction results were obtained, and most patients considered telemedicine consultation as good as face-to-face consultation in the specialties that were evaluated (most of them internal medicine). Despite these results, it is important to consider that there can be diseases that need to be treated in person by a medical provider.
Item number 12 evaluated the NPS index to know the fidelity of patients with the telemedicine service and its growth potential having a very good result. According to the literature, NPS rates >0 are considered good results, and rates >50% are excellent. The 43% obtained is close to the limit between good and excellent results, so it can be inferred that most patients are satisfied with the service and that a large percentage are so satisfied that they would even recommend it.
Conclusions
The scale developed was valid and reliable for the Hospital Clinica Nova population. However, due to the characteristics of the items, it may apply to other Spanish-speaking health organizations that use telemedicine for patient care. We are aware of other health systems in Mexico and Latin America that use this type of consultation, so we think this questionnaire could be helpful. Likewise, as it is a short questionnaire (nine items) with a duration ranging from 3 to 12 min, it is favorable for its application through telephone.17
Other tests for scale validation are described in the literature, such as criterion and construct validation; however, to perform these, comparison of the results with those obtained by another instrument designed to measure the same variable is required. For patient satisfaction with telemedicine consultations, as this is the first validated questionnaire in Spanish, other instruments will be required to perform this type of test.
The measurement of patient satisfaction is an essential part of evaluating the quality of clinical care. This indicator, along with clinical and economic results, allows knowing the effectiveness of health interventions such as telemedicine in the COVID-19 pandemic context.
Similarly, health satisfaction surveys must be developed and evaluated through validation and reliability tests. In this way, the results can be used more securely for decision making within organizations.
Limitations of the Study
The scale development, validation, and application were conducted during the COVID-19 pandemic. We cannot take these results as valid out of pandemic context.
Further Considerations
We consider that this instrument may be applicable in other Spanish-speaking counties, but it is important to test its validity in each population.
Authors’ Contributions
Conception and design of study were carried out by M.E.R.-I., J.I.T.-M., and C.E.T.-A. Acquisition of data was done by J.I.T.-M. and C.E.T.-A. Analysis and/or interpretation of data were done by M.E.R.-I. and J.I.T.-M. Drafting the article was by J.I.T.-M. Revising the article critically for important intellectual content was by M.E.R.-I. and C.E.T.-A. Approval of the version of the article to be published was done by M.E.R.-I., J.I.T.-M., and C.E.T.-A.
Disclosure Statement
No competing financial interests exist.
Funding Information
No funding was received for this article.
References
- 1. , Transmission of SARS-CoV-2: implications for infection prevention precautions. 2020. Available at https://www.who.int/news-room/commentaries/detail/transmission-of-sars-cov-2-implications-for-infection-prevention-precautions (last accessed
January 15, 2020 ). Google Scholar - 2. . 2020. Available at https://www.bbc.com/news/world-51839944 (last accessed
March 11, 2021 ). Google Scholar - 3. . 2021. Available at https://covid19.who.int (last accessed
January 15, 2021 ). Google Scholar - 4. Telemedicine in the face of the COVID-19 pandemic. Aten Primaria 2020;52:418–422. Crossref, Medline, Google Scholar
- 5. . Virtually perfect? Telemedicine for Covid-19. N Engl J Med 2020;382:1679–1681. Crossref, Medline, Google Scholar
- 6. . Implementing telemedicine in response to the COVID-19 pandemic. J Urol 2020;204:14–16. Crossref, Medline, Google Scholar
- 7. . Standards for reporting qualitative research: A synthesis of recommendations. Acad Med 2014;89:1245–1251. Crossref, Medline, Google Scholar
- 8. . Best practices for developing and validating scales for health, social, and behavioral research: A primer. Front Public Health 2018;6:149. Crossref, Medline, Google Scholar
- 9. . 2020. Available at https://iris.paho.org/bitstream/handle/10665.2/52244/PAHOIMSPHECOVID-19200028_eng?sequence=5 (last accessed
February 2, 2021 ). Google Scholar - 10. The impact of telehealthcare on the quality and safety of care: A systematic overview. PLoS One 2013;19;8:e71238. Crossref, Google Scholar
- 11. . e-Health in Mexico: Background, objectives, achievements, and challenges. Espacios Públicos 2012;15:65–94. Google Scholar
- 12. Service quality perceptions and patient satisfaction: A study of hospitals in a developing country. Soc Sci Med 2001;52:1359–1379. Crossref, Medline, Google Scholar
- 13. Development of Servqual Model for the measurement of quality services in a publicity company. Expert help. Perspectivas 2014;34:181–209. Google Scholar
- 14. Improvement of the quality of care through telemedicine and telecare. Madrid, Spain: Diaz de Santos, 2014. Google Scholar
- 15.
Institute of Medicine (US) Committee on Quality of Health Care in America . Crossing the Quality Chasm. Washington, DC, United States: National Academies Press, 2001. Google Scholar - 16. . Patient satisfaction with the healthcare system: Assessing the impact of socio-economic and healthcare provision factors. BMC Health Serv Res 2016;16:94. Crossref, Medline, Google Scholar
- 17. . Sampling telephone numbers and adults, interview length, and weighting in the California health interview survey cell phone pilot study. Public Opin Q 2007;71:793–813. Crossref, Google Scholar