Figures
Abstract
Background
In the context of a randomized controlled trial (RCT) on HIV testing in the emergency department (ED) setting, we evaluated preferences for survey modality and data quality arising from each modality.
Methods
Enrolled participants were offered the choice of answering a survey via audio computer assisted self-interview (ACASI) or pen and paper self-administered questionnaire (SAQ). We evaluated factors influencing choice of survey modality. We defined unusable data for a particular survey domain as answering fewer than 75% of the questions in the domain. We then compared ACASI and SAQ with respect to unusable data for domains that address sensitive topics.
Results
Of 758 enrolled ED patients, 218 (29%) chose ACASI, 343 chose SAQ (45%) and 197 (26%) opted not to complete either. Results of the log-binomial regression indicated that older (RR = 1.08 per decade) and less educated participants (RR = 1.25) were more likely to choose SAQ over ACASI. ACASI yielded substantially less unusable data than SAQ.
Citation: Reichmann WM, Losina E, Seage GR III, Arbelaez C, Safren SA, Katz JN, et al. (2010) Does Modality of Survey Administration Impact Data Quality: Audio Computer Assisted Self Interview (ACASI) Versus Self-Administered Pen and Paper? PLoS ONE 5(1): e8728. https://doi.org/10.1371/journal.pone.0008728
Editor: Rupert Kaul, University of Toronto, Canada
Received: August 6, 2009; Accepted: December 21, 2009; Published: January 15, 2010
Copyright: © 2010 Reichmann et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported by the National Institute of Mental Health [R01 MH073445, R01 MH65869]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Audio computer assisted self-interview (ACASI) is a survey modality that is frequently used in healthcare and clinical research settings to collect patient information. It is most useful when sensitive information is requested and subjects perceive that they may be judged by their responses [1]–[13]. Randomized studies of ACASI have observed that people are more likely to report “risky” behaviors, such as unprotected sex and needle sharing, in ACASI based surveys than in face-to-face personal interviews [1], [2], [6], [7], [12]. When ACASI has been compared with self-administered questionnaires (SAQ), the results have been mixed [5], [9], [10]. Some studies document no increased reporting of risky behaviors [5], while others have found an increased reporting of risky behaviors with ACASI [9]. Others have found that gender may modify the effect of survey modality on the amount of reporting of risky behaviors [10]. None of these studies have compared ACASI and SAQ in the emergency department (ED) setting.
Within the context of a randomized controlled trial (RCT) of HIV screening programs in the ED, we requested from subjects sensitive information via ACASI. However, in the first two months of our trial many participants requested to complete our survey via the more traditional pen and paper self-administered questionnaire (SAQ). Given this experience and the paucity of literature examining subject preferences and survey completeness when both SAQ and ACASI are offered, our objective was to more formally analyze differences in preferences and survey completion. Accordingly, we first sought to identify socio-demographic factors that might affect subjects' preference for SAQ vs. ACASI. Second, we sought to evaluate the impact of survey modality on the completeness of the data across different domains that address sensitive information.
Methods
Study Overview
The Universal Screening for HIV in the Emergency Room (USHER) trial was a randomized controlled trial designed to evaluate alternative approaches to routine HIV testing in the ED (ClinicalTrials.Gov: #NCT00502944) [14]. Within the context of this study, enrolled participants were asked to complete an 85-item survey that assessed their attitudes toward HIV testing and their knowledge about HIV infection [15], and inquired about their history of HIV testing, sexual behavior, mental health [16]–[18], alcohol [19], and drug use. Where possible the survey used standardized, validated items and scales [15]–[19]. Participants who were enrolled from April 1, 2007 to August 16, 2007 were offered the choice of completing the research survey via the ACASI or SAQ. The consent form for the study was uniformly standardized and did not contain information on survey modality.
The study randomized participants to a rapid oral HIV testing to be offered and conducted by a member of their health care team (provider arm) or by an independent HIV counselor (counselor arm) [20]. Randomization occurred after the participant agreed to enrollment. Upon randomization, but generally prior to HIV test offer, the participant was asked his/her preference for survey modality; the survey was provided according to patients stated preference. Minimizing disruptions in patient ED flow was critical for study success; on occasion, therefore, participants may have been interrupted during the survey so that the HIV screening test – which takes 20 minutes to develop – could be offered and conducted. The Brigham and Women's Hospital institutional review board approved the study, and the study was overseen by a data safety monitoring board.
Study Setting and Population
The Brigham and Women's Hospital ED is a major level 1 trauma center with over 56,000 patient visits per year. To be eligible for participation in the USHER trial, patients had to be: 1) from 18 to 75 years old, 2) fluent in English or Spanish, 3) assigned an emergency severity index (ESI) score of 3, 4, or 5 (range: 1–5; 1 = most severe, 5 = least severe) [21]–[23], 4) not known to be HIV-infected, and 5) not be in pre-natal care. Enrolled participants provided written informed consent and were not paid for participation.
Survey Development
Survey questions were developed at a 4th grade standard reading level and were reviewed by a literacy expert for clarity and comprehensibility. Both survey modalities asked identical questions presented in the same order.
The ACASI program was developed using the Questionnaire Development System (QDS™) software (NOVA Research Company). During the enrollment process, participants were told that a laptop would be wheeled into their room if they were to choose the computer version of the survey. Participants were instructed on how to use the laptop and that no prior computer experience was necessary for its completion. Research assistants were available for assistance in completing the survey at any time. The ACASI survey used in the USHER trial was available to participants in both English and Spanish, each in a male or female voice (four different combinations); participants completing the survey via ACASI had their choice among the four versions. A single question appeared on the screen at a time, and the responses were highlighted as they were read. Participants who chose ACASI completed the survey on a portable laptop on a bedside table in their ED room. Participants had the option to pause and resume the ACASI program as needed during their ED stay. They could also skip questions or go back and change their answer to previously answered questions. Automated skips were built into the program where appropriate to shorten the length of time needed to complete the survey. For example, if the participants answered to ‘No’ to being a current smoker, the ACASI program would skip the question about the number of cigarettes smoked per day.
The SAQ version of the survey was also available to participants in both English and Spanish. The 18-page survey allowed participants to mark their response by checking a box. Participants could pause and skip questions as needed. The automated skips that were built into the ACASI program were written on the page. For example, if the participant responded ‘No’ to the smoking question, he/she would see a note that instructed to advancement to the next applicable question.
Data Elements
Demographic data, including age, gender, race/ethnicity, primary language, and education were collected from the participants by the research assistant at the time they were offered enrollment, which was prior to being offered the survey. The survey collected data on seven domains, which included, attitudes toward HIV testing; knowledge about HIV [15]; socioeconomic status and medical history; history of HIV testing; sexual behavior, including the number of different sexual partners in the past 12 months and the past 3 months; mental health [16]–[18]; and smoking status, alcohol usage [19], and illicit drug usage. Data on HIV knowledge was collected using the validated 18-item HIV Knowledge Questionnaire, which has been found to be internally consistent and suitable for low-literacy populations [15]. Mental health was assessed using the validated Center for Epidemiological Studies-Depression (CES-D) scale, used to screen for depressive symptoms. The CES-D scale is a validated survey that has shown good measurement properties (Cronbach's alpha = 0.9) [16]–[18]. Alcohol usage was evaluated with the validated Alcohol Use Disorders Test (AUDIT), developed by the World Health Organization [19]. Further details regarding the data elements can be found in Appendix S1.
Defining Unusable Data in Survey Response
We defined the data for a specific domain, such as sexual behavior or depression, as ‘unusable’ if the respondent completed less than 75% of the questions in that particular domain. We chose a 75% threshold because it was the minimum to allow a validated score to be constructed based on the data provided by a particular respondent. We used 90% as an alternative threshold in a sensitivity analysis.
Statistical Analysis
All analyses were conducted using SAS version 9.1 (Cary, NC). Means and standard deviations were calculated to describe continuous variables; proportions were calculated for categorical variables. Log-binomial regression was performed to calculate both unadjusted and adjusted relative risks (RR) of choosing the SAQ version of the survey over the ACASI. We chose log-binomial regression because it provides better estimates of the RR than does standard logistic regression when the prevalence of the outcome is greater than 10%. Ninety-five percent confidence intervals were constructed for both unadjusted and adjusted RRs.
The analysis examining the impact of survey modality on the magnitude of unusable data within across four specific domains (sexual history; depression; alcohol use; and illicit drug use) contained several steps. First, every subject was assigned a propensity score, defined as the probability of choosing the SAQ version of the survey given a set of covariates, including age, gender, race, language, education, religion, income, employment status, comorbidity, and health care utilization. We used the propensity score to adjust for multiple confounders [24]. The next step included building a log-binomial regression model to evaluate the impact of survey modality on the amount of unusable data (less than 75% complete for a specific domain), after adjusting for the propensity score. We divided the propensity score into quintiles and adjusted for the quintile of the propensity score in the log-binomial regression model that evaluated the impact of survey modality on the amount of unusable data. As a sensitivity analysis, we repeated the above analyses using 90% as the threshold for unusable data.
Results
Sample
From April 1 to August 16, 2007, 758 patients were enrolled in USHER Trial when both the ACASI and SAQ were offered. Of those enrolled, 561 (74%) patients agreed to answer the survey; 218 (39%) chose the ACASI, while 343 (61%) chose the SAQ.
The mean age of all participants enrolled was 37 years (standard deviation of 13). Two hundred ninety-three (39%) were non-Hispanic white, 147 (20%) were non-Hispanic black, 231 (31%) were Hispanic, and 72 (10%) were classified as other. A majority of the participants were female (64%) and reported English as their primary language (73%). Two hundred ninety (38%) participants had a high school degree or less, while 262 (35%) had at least a college degree. The demographic features were examined by survey completion mode in Table 1.
Associations with Survey Modality
Demographic variables (age, race, and education) were associated with survey modality, while other self-reported psychosocial variables were not (Table 2). Results of the multivariable log-binomial regression confirmed the findings of the unadjusted associations (Table 2). The multivariable log-binomial regression showed that participants who were older (RR = 1.08; 95% CI: 1.03, 1.13; per 10-year increase in age), non-white (1.26; 95% CI: 1.08, 1.47), and having a highest education attainment of high school degree or less (RR = 1.25; 95% CI: 1.09, 1.43) were more likely to choose the paper SAQ over the ACASI version of the survey (Table 2).
Impact of Survey Modality on Unusable Data
Results examining the impact of survey modality on the magnitude of unusable data were examined across four different domains (sexual history; depression; alcohol use; and illicit drug use) are displayed in Figure 1 and Table 3. Figure 1 displays the proportion of unusable data by section of the survey for our base case threshold of unusable data of 75%. For the sexual history and depression domains the proportion of unusable data were 25% and 25% for the SAQ modality compared to 13% and 17% for the ACASI modality respectively. The smallest difference between the two modalities occurred in the alcohol use domain. Twenty-four percent of the participants who chose ACASI had unusable data for this domain compared to 30% for those who chose SAQ. The greatest disparity between the two modalities with respect to unusable data was for the illicit drug use domain with 18% of ACASI participants having unusable data compared to 36% of the SAQ participants.
Unusable data was defined as having responded to less than 75% of the questions for the selected domains of the survey. The dark gray bars represent participants who chose SAQ, while the white bars represent participants who chose the ACASI version of the survey. Error bars in Figure 1 represent the 95% confidence interval. 95% confidence intervals that do not overlap imply a statistical difference in unusable data by survey modality for that particular domain. CES-D: Center for Epidemiologic Studies Depression Scale. AUDIT: Alcohol Use Disorders Identification. STD: Sexually Transmitted Disease. ACASI: Audio Computer Assisted Self-Interview. SAQ: Self-Administered Questionnaire.
Table 3 displays relative risks having unusable data for the four domains at the base case threshold of 75%. The results of an unadjusted analysis and an analysis that adjusts for the quintile of the propensity score are displayed. In all cases the estimated RR was greater than 1, indicating that participants who chose the SAQ version of the survey were more likely to have unusable data in each of the four domains than those who chose the ACASI version of the survey. In the unadjusted analysis this finding was statistically significant in the sexual behavior, depression, and illicit drug use domains. The finding was not statistically significant for the alcohol use domain. In the analysis that adjusted for the quintile of the propensity score the finding was statistically significant in the depression and illicit drug use domains but not in the sexual behavior or alcohol use domain (Table 3).
Results were similar for our sensitivity analysis using 90% as the threshold for unusable data. The proportion of unusable data for the ACASI version of the survey was 13%, 18%, 26%, and 19% for the sexual behavior, depression, alcohol use, and illicit drug use domains, respectively. This proportion was higher for the SAQ version of the survey with 28%, 28%, 31%, and 38% unusable data across the four domains, respectively. When we adjusted for the quintile of the propensity score, survey modality significantly impacted data quality in the sexual behavior, depression, and illicit drug use domains, but not the alcohol use domain.
Discussion
ACASI is a popular mode of data collection in clinical research settings, especially when the survey includes sensitive questions. In the context of a randomized trial, we initially only offered the ACASI version of our survey, but anecdotal experience suggested that participants may prefer to complete the survey via SAQ. Thus, participants were provided the option of completing our survey via ACASI or the more traditional pen and paper SAQ. We provided this option so that we could more formally address the question of participant preference of survey type and the data quality arising from each modality. In the context of randomized trial of universal screening for HIV in the emergency department, we found that patient age, race, and education were associated with patient preference for survey modality. For every 10-year increase in age, patients were 8% more likely to favor the SAQ version over ACASI. Non-white participants were 26% more likely to choose SAQ over ACASI and those with a high school degree or less were 25% more likely to choose the SAQ over ACASI. Because surveys were offered prior to testing, HIV rapid test results should have little influence on survey preference.
Despite the benefit of increased preference for the SAQ version of the survey, using this modality had its tradeoffs. Adjusted relative risks of having more unusable data with the SAQ, as opposed to ACASI, ranged from 1.3 (95% CI: 0.9, 1.8) (alcohol consumption domain) to 2.0 (95% CI: 1.4, 2.9) (illicit drug use domain), indicating that the SAQ version of the survey yielded more unusable data. We found that the SAQ version yielded more unusable data than the ACASI version across all four domains assessing sensitive information. This finding was robust regardless of the threshold that was used for defining unusable data or adjustments for the propensity score, a method of adjusting for multiple potential confounders [24]. The implications of these findings are substantial. The proportion of participants with unusable data ranged from 25 to 36% among those who chose the SAQ when the threshold for having unusable data was less than 75% complete. This relationship existed across critical domains ranging from sexual risk, alcohol and drug use to depression. Incomplete data makes it challenging, if not impossible, for researchers to use this information.
When offered the choice of answering our survey via SAQ or ACASI, we found that patients were more likely to choose the SAQ. These findings were especially true among less educated and older participants. However, this is the only study to our knowledge that examined patient choice of survey modality in the ED. It is possible that the laptop which delivered the ACASI proved to be too cumbersome for our participants and that the use of a handheld device may be more efficient and acceptable in this setting. However, we specifically chose the laptop to avoid the disappearance (theft) of devices, which we felt was reasonable given the busy ED setting. We also found that ACASI did provide the most complete data, as other studies have found [2], [6]–[9].
There are several limitations to our study. First, the primary objective of our analysis, while examined within the context of a RCT, was not assessed with a randomized design. It is possible that patients who chose the SAQ would have accepted the ACASI if no other alternative was given to them and vice versa. Also, we were unable to control for unmeasured confounders that might obscure the association between patient preference and education. Such measures for which we were not able to control include reading and computer literacy. While these measures are likely correlated with level of education, it may be important to understand how these two factors jointly affect the choice of survey modality. Another limitation of our study is that exit interviews were not conducted, which might have provided insights into barriers of ACASI administration in the ED. Lastly, two of the domains (sexual behavior and illicit drug use) in which we looked at the impact of survey modality on data quality were just a group of questions about specific topic and were not measured using validated scales. Despite these limitations, our study is unique in its setting, as no other study has compared ACASI and SAQ in the ED where an increasing number of patients are being tested for HIV [25]–[30].
We found that age, race, and education play an important role in survey preference. While more patients preferred the SAQ version of our survey over the ACASI, the ACASI version provided more complete data with regard to sections that cover sensitive topics. Further research should assess how strong the preference for SAQ over ACASI is using a randomized design. Also, research on novel approaches that maximize ED patients' comfort with ACASI would be of great benefit because the amount of missing data in the SAQ may limit its utility.
Supporting Information
Appendix S1.
This document is the paper version of the survey.
https://doi.org/10.1371/journal.pone.0008728.s001
(0.21 MB DOC)
Acknowledgments
The authors would like to acknowledge the co-investigators and staff of the Universal Screening for HIV in the Emergency Room trial including: Carrie Braverman, Amy Louise Case, Laurel Donnell-Fink, MPH, Susan Larrabee, Regina Mikulinsky, A. David Paltiel, PhD, Mary Pisculli, MD, MPH, Mariesa Ricks, Paul E. Sax, MD, and Ron Walls, MD. We would like to thank the faculty and staff of the Brigham and Women's Hospital Emergency Department for their participation in the USHER Trial and the patients for their time in participating in our study.
Author Contributions
Conceived and designed the experiments: WR EL GRS CA SS JK AH RPW. Performed the experiments: WR AH. Analyzed the data: WR EL GRS CA SS JK RPW. Contributed reagents/materials/analysis tools: WR EL GRS CA SS JK AH RPW. Wrote the paper: WR. Critical revision of the manuscript and data: EL GRS CA SS JK RPW. Critical revision of the manuscript: AH.
References
- 1. Macalino GE, Celentano DD, Latkin C, Strathdee SA, Vlahov D (2002) Risk behaviors by audio computer-assisted self-interviews among HIV-seropositive and HIV-seronegative injection drug users. AIDS Educ Prev 14: 367–378.
- 2. Des Jarlais DC, Paone D, Milliken J, Turner CF, Miller H, et al. (1999) Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. Lancet 353: 1657–1661.
- 3. Gribble JN, Miller HG, Rogers SM, Turner CF (1999) Interview mode and measurement of sexual behaviors: Methodological issues. Journal of Sex Research 36: 16–24.
- 4. Jones R (2003) Survey data collection using Audio Computer Assisted Self-Interview. West J Nurs Res 25: 349–358.
- 5. Morrison-Beedy D, Carey MP, Tu X (2006) Accuracy of audio computer-assisted self-interviewing (ACASI) and self-administered questionnaires for the assessment of sexual behavior. AIDS Behav 10: 541–552.
- 6. Rogers SM, Willis G, Al-Tayyib A, Villarroel MA, Turner CF, et al. (2005) Audio computer assisted interviewing to measure HIV risk behaviours in a clinic population. Sex Transm Infect 81: 501–507.
- 7. Simoes AA, Bastos FI, Moreira RI, Lynch KG, Metzger DS (2006) A randomized trial of audio computer and in-person interview to assess HIV risk among drug and alcohol users in Rio De Janeiro, Brazil. J Subst Abuse Treat 30: 237–243.
- 8. Tourangeau R, Smith TW (1996) Asking sensitive questions: The impact of data collection mode, question format, and question context. The Public Opinion Quarterly 60: 275–304.
- 9. Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, et al. (1998) Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science 280: 867–873.
- 10. Webb PM, Zimet GD, Fortenberry JD, Blythe MJ (1999) Comparability of a computer-assisted versus written method for collecting health behavior information from adolescent patients. J Adolesc Health 24: 383–388.
- 11. Rickert VI, Davison LL, Breitbart V, Jones K, Palmetto NP, et al. (2009) A randomized trial of screening for relationship violence in young women. J Adolesc Health 45: 163–170.
- 12. Ghanem KG, Hutton HE, Zenilman JM, Zimba R, Erbelding EJ (2005) Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sex Transm Infect 81: 421–425.
- 13. Kurth AE, Martin DP, Golden MR, Weiss NS, Heagerty PJ, et al. (2004) A comparison between audio computer-assisted self-interviews and clinician interviews for obtaining the sexual history. Sex Transm Dis 31: 719–726.
- 14.
(NIMH) NIoMH, editor(NIMH) NIoMH Two approaches to routine HIV testing in a hospital emergency department. National Institutes of Mental Health (NIMH)..
- 15. Carey MP, Schroder KE (2002) Development and psychometric evaluation of the brief HIV Knowledge Questionnaire. AIDS Educ Prev 14: 172–182.
- 16.
U.S. Department of Health and Human Services (2004) CES-D Scale. Bethesda: Department of Health and Human Services & National Institute of Health.
- 17. Radloff LS (1997) The CES-D scale: a self-report depression scale for research in the general population. Applied Psychological Measurement 1: 385–401.
- 18. Hann D, Winter K, Jacobsen P (1999) Measurement of depressive symptoms in cancer patients: evaluation of the Center for Epidemiological Studies Depression Scale (CES-D). Journal of Psychosomatic Research 46: 437–443.
- 19.
Barbor TF, Higgins-Biddle JC, Saunders JB, Montero MG (2001) The alcohol use disorders identification test (AUDIT): Guidelines for use in primary care. 2nd edition. Geneva: World Health Organization.
- 20.
Walensky RPAC, Reichmann WM, Wright EA, Katz JN, Hare AQ, et al. (2009) Counselor- versus Provider-Based HIV Testing in the Emergency Department: Results from the Universal Screening for HIV in the Emergency Room Trial. CROI 1039.
- 21. Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N (2000) Reliability and validity of a new five-level triage instrument. Acad Emerg Med 7: 236–242.
- 22. Wuerz RC, Travers D, Gilboy N, Eitel DR, Rosenau A, et al. (2001) Implementation and refinement of the emergency severity index. Acad Emerg Med 8: 170–176.
- 23. Eitel DR, Travers DA, Rosenau AM, Gilboy N, Wuerz RC (2003) The emergency severity index triage algorithm version 2 is reliable and valid. Acad Emerg Med 10: 1070–1080.
- 24. D'Agostino RB Jr (1998) Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 17: 2265–2281.
- 25. Centers for Disease Control and Prevention (CDC) (2007) Rapid HIV testing in emergency departments—three U.S. sites, January 2005–March 2006. MMWR Morb Mortal Wkly Rep 56: 597–601.
- 26. Brown J, Shesser R, Simon G, Bahn M, Czarnogorski M, et al. (2007) Routine HIV screening in the emergency department using the new US Centers for Disease Control and Prevention Guidelines: results from a high-prevalence area. J Acquir Immune Defic Syndr 46: 395–401.
- 27. Haukoos JS, Hopkins E, Eliopoulos VT, Byyny RL, Laperriere KA, et al. (2007) Development and implementation of a model to improve identification of patients infected with HIV using diagnostic rapid testing in the emergency department. Acad Emerg Med 14: 1149–1157.
- 28. Lyss SB, Branson BM, Kroc KA, Couture EF, Newman DR, et al. (2007) Detecting unsuspected HIV infection with a rapid whole-blood HIV test in an urban emergency department. J Acquir Immune Defic Syndr 44: 435–442.
- 29. Mehta SD, Hall J, Lyss SB, Skolnik PR, Pealer LN, et al. (2007) Adult and pediatric emergency department sexually transmitted disease and HIV screening: programmatic overview and outcomes. Acad Emerg Med 14: 250–258.
- 30. Silva A, Glick NR, Lyss SB, Hutchinson AB, Gift TL, et al. (2007) Implementing an HIV and sexually transmitted disease screening program in an emergency department. Ann Emerg Med 49: 564–572.