Translation, Adaptation, and Validation of the Filipino Version of the Telehealth Usability Questionnaire (TUQ-F)

Introduction

The World Health Organization (WHO) had declared on March 11, 2020 the outbreak of the COVID-19 pandemic. Because of this, healthcare services had to shift from set standard practices all over the world. Hospitals and clinics have implemented different protocols to provide proper health care, including the drastic shift to telehealth.

The World Health Organization (WHO)[1] defined telehealth as a means of delivery for healthcare services, with distance being a crucial factor. Moreover, health care providers use various modes of telecommunication technology and strategies to exchange valid information for evaluation and diagnosis, treatment and prevention, and research of diseases and injuries. Consequently, this provides as an excellent medium for medical education and healthcare improvement.

Telehealth uses telecommunication technology to provide long-distance healthcare, education and information[2]. The methods used in telehealth services include telephone calls, emails, the Internet, video calls or conferences, and remote devices. Telemedicine, usually used interchangeably with telehealth, is under telehealth and refers specifically to clinical services[3]. There has been a significant increase in using telehealth services to address the health care needs since the COVID-19 outbreak[4]. 

In the Philippines, the Department of Health acknowledges telemedicine and promotes its use to avert the surge of patients leading to higher risks posed by unnecessary crowding in hospitals. In line with this, a telemedicine website was launched together with HealthNow in May 2020, where Filipinos can consult about their illness free of charge, whether it is related to COVID-19 or not. Similar platforms have also started telemedicine services in the country, including Medgate, KonsultaMD, CloudPx, and SeeYouDoc [5].

Yip, et al.[6] formulated a 15-item Telemedicine Satisfaction Questionnaire (TSQ) during the advent of telemedicine. However, a review of questionnaires intended to measure patient satisfaction developed in the 2000s, including the TSQ, has been found to focus only on three aspects of usability: usefulness, satisfaction and interaction quality. Other questionnaires cited in the study included the Telemedicine Patient Questionnaire (TMPQ) and Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) [7,8].

As such a gap exists in comprehensively assessing different aspects of usability in the telemedicine consultation experience, Parmanto, et al.[9] formulated the Telehealth Usability Questionnaire in 2016. The TUQ is a valid and reliable tool that comprises 21 questions sourced primarily from the TSQ, combined with questions found in previously mentioned patient satisfaction in telemedicine questionnaires. Moreover, Parmanto, et al.[9] had defined usability as the extent that a product provides for preset users in achieving specific goals with effectiveness, efficiency and satisfaction in a predetermined use. The usability factors included in assessing telemedicine are: usefulness, ease of use, effectiveness, reliability and satisfaction. At present, no version of this telehealth questionnaire has been translated and validated in the Filipino language for local use. 

The primary objective of this study was to adapt a standard tool, from English to Filipino, by evaluating its validity, reliability and equivalence. The final translated tool aims to assess the usability of telehealth services in the local (Philippine) setting, with the prospect that resulting information will directly improve policy decisions regarding telemedicine within hospital systems.

Methodology

Conceptual Framework

The theoretical framework of research focuses on three main components: 1) the patient and medical professionals, identified as those who utilize telehealth services under the University of Santo Tomas Hospital (USTH); 2) the telehealth system used in USTH; 3) the translated Filipino version of the Telehealth Usability Questionnaire (TUQ-F). Within the framework, independent variables are domains or usability factors that may affect the patients' level of satisfaction: usefulness, ease of use and learnability, effectiveness, reliability, satisfaction and future use. The dependent variable in the study would be usability of the telehealth system of USTH.

Study Population and Setting

The study was conducted in a tertiary, non-stock, non-profit private hospital located in Manila, Philippines, and was accomplished for 10 months between April 2021 and February 2022. Data collection utilized both common and available online platforms accessible on various devices. Google Forms was used to administer the questionnaire and Google Meet was used to facilitate cognitive debriefing for interviewed participants.

The study population comprised patients who have utilized telehealth services under the University of Santo Tomas Hospital (USTH). The criteria for selection included those who are: (1) literate and bilingual, (2) without any critical conditions that would limit their participation, (3) without any conditions that would limit their understanding and comprehension, and (4) with telehealth experience in the past six months. The study only accommodated those of legal age (18 years old and above) and excluded senior citizens (>60 years old) and minors (<18 years old). Termination criteria included participants’ withdrawal from the study in the middle of answering the tool and refusal to answer item/s from the questionnaire. Convenience sampling was used in acquiring participants for both pilot and field testing within the target population. The researchers collaborated with several departments who provide telehealth, and participants were provided a link of the questionnaire after their consultation. 

Study Design

The study followed a translation validation design aimed at adapting a validated and reliable tool that measures usability of telehealth services from the original language, English, to the target language － Filipino. This study was divided into four phases (Figure 1):

• Phase 1. Translation of original text into the target language: Prior to commencement of the study, permission to use the TUQ was sought and granted by the original author. University of Santo Tomas - Sentro sa Salin at Araling Salin aided in translating the existing tool for the intended population. Two forward translations were performed independently by separate individuals and were reconciled to resolve any discrepancies, give an alternative translation if necessary, and, when appropriate, ensure linguistic compatibility. This reconciliation process involved one native speaker or language professional of the target language who was not involved with forward translation. Lastly, a focus group discussion (FGD) was done, which allowed in-depth discourse on the translated tool and provided an opportunity to recognize what needed to be improved.
• Phase 2. Employment of experts to review the translation: The primary purpose of independent review was to analyze the target-language translation in light of all the information provided for the intended population. A minimum of three to twenty is generally recommended for the expert panelists[10,11]. The study sought expertise of five bilingual health professionals whose services include the use of telehealth. These health professionals independently analyzed the preliminary version of the translated tool, assessed validity, and provided commentary or alternative translation if deemed necessary.
• Phase 3. Pretesting (cognitive debriefing and pilot study): In the finalization process, the pre-final Filipino version of the TUQ was subjected to a pilot test that aimed to increase research quality and strengthen the tool’s reliability[12]. Thirty telehealth patients who passed the study’s selection criteria participated in pilot testing, and three voluntary participants from this population underwent additional cognitive debriefing to identify items or parts of the questionnaire that needed revisions.
• Phase 4. Field testing: To assess equivalence of the translated TUQ-Filipino (TUQ-F), and satisfy the standard set by the original objective of the tool[13], field testing of the TUQ-F was conducted. Both the original English TUQ and translated TUQ-F were administered to 85 patients from different departments of USTH who satisfied the selection criteria[14]. The administration of questionnaires followed the same procedures as done previously in pilot testing.

Ethical Considerations

The study was approved by the Institutional Review Board of the University of Santo Tomas Hospital (USTH-IRB), and the research abides by the National Ethical Guidelines for Health & Health-Related Researches (NEGHHRR) 2017. To address concerns on privacy, the study complies with the Data Privacy Act of 2012 and its implemented rules and regulations in 2016.

Research Instrument

A 4-part survey, including patient demographics, telehealth profile, the TUQ-F and the original English TUQ was conducted during field testing. The patient demographics included age, gender, marital status, educational attainment, employment status and location of participants. Information pertaining to telehealth includes the nature of consultation, patient’s access to telehealth contact, internet access and device utilized. The TUQ is a 21-item assessment that evaluates five subscales, or usability factors, of telemedicine. This includes usefulness, ease of use, effectiveness, reliability, satisfaction, and future use[9]. It uses ordinal-level measurement wherein categories are ranked based on their agreement/disagreement with the statement regarding telehealth usability. The tool is brief, and a simple mean and SD scoring algorithm provides a scored value for each item which can then be used to examine usability of telehealth systems[9]. The higher the overall average, higher the usability. 

Statistical Analyses

All analyses were conducted using SPSS. Descriptive statistics used in the study are mean, standard deviation, frequency and percentage. Content validity was evaluated with CVR, I-CVI and S-CVI/Ave, having the following cut-off values: 1.00, 0.80 and 0.90, respectively. While the inter-rater reliability was assessed using the Fleiss Kappa method with a cut-off value of 0.75[15,16], reliability coefficients used Cronbach’s alpha in evaluation with a cut-off value of 0.70[9,17]. Equivalences were also determined between the English and Filipino versions. Since the tool uses ordinal-level measurement, a non-parametric test like the Wilcoxon-rank sum test was utilized for the linguistic equivalence[18]. A significance level of 0.05, with the hypothesis that there is no significant difference between the mean score of the original TUQ and TUQ-Filipino, was used to interpret data. Kappa test for concordance and marginal homogeneity test was also employed to determine the level and presence of agreement[19,20]. Furthermore, Bland-Altman (B&A) plot analysis was utilized to illustrate the degree of agreement between each domain[21,22]. Lastly, conceptual equivalence for each respective subscale was observed through Spearman’s rank-order correlation coefficient[23]. 

Results

Demographic and Telehealth Profile

The demographics and telehealth profile of the study (Table 1) demonstrates that majority of the respondents were female (69.4%), single (75.29%), had a college/university degree (71.64%) and are current students (57.65%). Most reside in NCR (57.65%), while the mean age of respondents was 29.14 years old (SD = 10.57) at the time of writing. Nature of consultation was mostly first consultation (70.59%), with most having consulted at the Internal Medicine department (32.94%). Access to the internet was mainly through Wi-Fi (92.94%), and most used cell phones (72.94%) for their telehealth consultation. Lastly, they were able to know the hospital’s telehealth services through online resources (56.47%) such as websites and social media pages.

Content Validity

After expert panel evaluation and discussion, the computed scores were 1.00 for essentiality, 0.94 for relevance and 0.92 for clarity (Table 2). Obtained Fleiss Kappa scores were >0.75. All translated items were retained, and several statements were modified to fit the cultural context through an FGD (Table 3).

Internal Consistency

Reliability coefficients were determined using Cronbach’s alpha after the pilot testing and cognitive debriefing interviews. Values obtained for each of the five domains and for the whole TUQ were 0.965, 0.985, 0.971, 0.990, 0.984 and 0.993, respectively. These scores were able to meet the cut-off score of 0.70.

Linguistic Equivalence

The mean, standard deviation and mean difference scores for each item and domain between TUQ-F and TUQ were tabulated (Table 4). All the mean differences were less than ±0.25.

The computed Wilcoxon Signed Rank p-values for all items and domains ranged from 0.06-1.0, which are all above 0.05, denoting no significant differences between the mean scores of the TUQ and TUQ-F. The computed Kappa range for each item was 0.58-0.83, which signified moderate to almost perfect agreements between the two versions. Results of the marginal homogeneity test on majority of items and each domain were not statistically different (p>0.05), except for item 10, which means that both TUQ-F and TUQ are the same and homogenous (Table 5).

The Bland-Altman analysis (Figure 2) showed mean difference scores (95% confidence interval, CI) of 0.03 (-0.715 to 0.770), -0.02 (-0.647 to 0.612), 0.02 (-0.642 to 0.687), 0.10 (-0.047 to 0.381) and 0.03 (-0.092 to 0.334) for the five domains, respectively; with corresponding p-value of the predictor (mean) values of 0.112, 0.286, 0.154, 0.322 and 0.763. The majority of scatterplots for each dimension of the Bland-Altman plots were within the upper and lower boundaries indicating agreement between the two versions.

Conceptual Equivalence

The Spearman’s rank-order correlation coefficient per item ranged from 0.613-0.87, while coefficients of each domain ranged from 0.863-0.922 (Table 6). All correlation coefficients are statistically significant at 0.01%.

Discussion

Validity and reliability are two fundamental elements in evaluating a measurement instrument. Validity is concerned with how an instrument measures what it is intended to measure. Reliability is concerned with the ability of an instrument to measure consistently. It should be noted that the reliability of an instrument is closely associated with its validity. An instrument cannot be valid unless it is reliable[24].

Content validity of the translated TUQ was measured to assess whether the components of scale are able to cover all aspects of that which is being measured in a balanced way[25]. The content validity ratio (CVR) value for each item was calculated to assess inclusion or exclusion within the resulting translated tool. Expert panel evaluation for each of the 21 items fulfilled the computed cut-off value to justify inclusion in the translated TUQ-Filipino[15, 26]. Similarly, each item had their item-level content validity index (I-CVI) and overall scale-level content validity index (S-CVI/Ave) determined to evaluate clarity and relevance of items within the instrument[15]. Each item in the translated tool met the cut-off scores suggesting clarity and appropriateness or relevance[27]. In terms of inter-rater reliability, results showed excellent agreement[28] among the expert panel evaluators.

In the process, the focus group discussion with experts resulted in modifying some of the translations within the TUQ-F to fit the Filipino context and to be easily understood by the intended respondents. Comments were integrated into the preliminary translation and developed into the final version of the TUQ-F. For example, in items 11, 12, 14 and 18, the term “manggagamot,” as translation of “clinician,” in the TUQ was replaced with the more colloquially used term, “doktor.”

Internal consistency of the whole TUQ-Filipino tool, as well as five domains were assessed through Cronbach’s coefficient alpha with a cut-off value of 70 (acceptable). All of the domains in the translated tool obtained a Cronbach’s coefficient alpha score of >0.90, indicating excellent internal consistency within each domain and within the whole tool[9, 17]. These results affirm that the TUQ-F is valid and reliable in evaluating Filipino patients’ telehealth experiences.

Tests for equivalence done on field testing results showed acceptable linguistic equivalence with the TUQ and TUQ-F having no statistically significant differences in mean differences at a ±0.25 cut-off, showing that TUQ-F was contextually valid and an acceptable equivalent translation[21] of the original tool. This was also supported by the Kappa test for concordance results for each of the 21 items which showed moderate to almost perfect agreement between the two versions of the TUQ at p<0.01[27]. 

To further analyze the level of agreement between the TUQ and TUQ-F, marginal homogeneity test was performed and showed that almost all items and each of the five domains lack significant difference, thus implying that both versions are homogenous, or essentially the same, with each other[20]; except for item 10. While additional testing to identify the cause of deviation was already outside the scope of this study, it can be theorized that such can happen because of limitations of language to where the original tool is being translated and limitations on the translation process itself. Further analysis, with Bland-Altman plots, supported the aforementioned data as well, with composite measure scores with no significant difference at a 95% confidence interval[21, 22]. Similarly, Spearman’s rank-order correlation coefficients per item and per domain showed that there was conceptual equivalence between TUQ and TUQ-F. This means that each item and domain were comparable with one another and evaluated the same constructs of telehealth experience in their corresponding contexts of use[21, 22].

In as much as all performed statistical analyses are concerned, the TUQ-F proved to be employable for the purposes intended. However, the tool is limited only to respondents adept at using the Filipino language. We recommend that TUQ-F be tested in other government and private hospitals or medical centers that employ telemedicine services using Filipino as their primary communication medium. The study was only conducted on a limited number of patients, and further research may consider increasing the sample size and range of study, including those using other telehealth platforms or those in other urban or rural regions who may have different modes of accessing telemedicine services.

Conclusion

This was the first study that translated and validated a telehealth usability questionnaire to Filipinos to the best of our knowledge. By and large, results showed that the 21-item Filipino version of TUQ had satisfactory content validity, internal consistency, and linguistic and conceptual equivalence. The TUQ-F, therefore, was a valid and appropriate instrument to evaluate usability of telehealth services in the local routine clinical practice, which may aid in improving process flows and policy decisions regarding telemedicine usability within hospital systems. Moreover, the questionnaire may serve as a medium for additional translation in major dialects within the Philippines (eg, Cebuano, Hiligaynon, Waray, etc.), which allows non-English-speaking Filipinos to impart their telemedicine experience to their health care professionals. The qualitative approach for translation studies may also be explored for future related research.

DISCLOSURE AND CONFLICT OF INTEREST

The researchers declare no form of conflict of interest, whether financial, familial, or proprietary, with their participants, validators or the study.

ACKNOWLEDGMENTS

The members of the research team would like to extend their gratitude to the following people for their contribution: Dr. Warren Bacorro, Sir Arturo Patungan, Jr., Dino Tordesillas, Ph.D., Dr. Justin Santos, Dr. Maria Philina Rita Pasicolan, Dr. Carina Callangan-Santiago, Dr. Estrellita Judan-Ruiz, Dr. Encarnita Ampil, Dr. Ma. Antonia Aurora Valencia, Dr. Norberto Martinez and Dr. Angelica Guzman-Hernandez. To our family and friends, we thank them for giving us their support and also helping us in this research in any way they could. And lastly, we would also want to thank God Almighty for guiding us and keeping us in His loving grace, for blessing us and making this possible to achieve.