Key Results Among the 459 participants, 22% reported unhealthy alcohol use and 25% reported drug use in the past year. The SISQ-alcohol had sensitivity of 73.3% (95% CI 65.3–80.3) and specificity of 84.7% (95% CI 80.2–88.5), AUC = 0.79 (95% CI 0.75–0.83), for detecting unhealthy alcohol use, and sensitivity of 86.7% (95% CI 75.4–94.1) and specificity of 74.2% (95% CI 69.6–78.4), AUC = 0.80 (95% CI 0.76–0.85), for alcohol use disorder. The SISQ-drug had sensitivity of 71.3% (95% CI 62.4–79.1) and specificity of 94.3% (95% CI 91.3–96.6), AUC = 0.83 (95% CI 0.79–0.87), for detecting unhealthy drug use, and sensitivity of 85.1 (95% CI 75.0–92.3) and specificity of 88.6% (95% CI 85.0–91.6), AUC = 0.87 (95% CI 0.83–0.91), for drug use disorder. INTRODUCTION Screening followed by brief intervention for unhealthy alcohol use in adult primary care patients is recommended by the United States Preventive Services Task Force, and is among the most cost-effective prevention services. – Screening, brief intervention, and referral to treatment (SBIRT) programs, which address drug use as well as alcohol, have been widely promoted and disseminated with the support of federal agencies. – While the efficacy of SBIRT for reducing drug use in U.S. Populations has not been clearly established, – screening in medical settings may be justified on clinical grounds.
These include the impact of drug use on prevention and treatment of other medical conditions, drug–medication interactions, effects on adherence, risk of prescription opioid overdose, and potential to improve health-related quality of life. Nonetheless, screening - even for alcohol alone - has proven difficult to implement and sustain in regular primary care practice. – In recent years, some implementation barriers have been eased by the development of validated single-item screening questions (SISQs) to identify unhealthy use of alcohol and drugs., The SISQs are brief enough to be easily incorporated into time-pressured practice settings, and have demonstrated high sensitivity and specificity for identification of unhealthy use in primary care patients. A recent study indicates that the SISQs may even be able to accurately identify and distinguish substance dependence. SISQs in studies to date have been asked by an interviewer, and this approach has limitations in some practice settings. Interviewer-administered questions can lose their fidelity when administered outside the research context, even by trained staff., The interviewer-administered approach may also be difficult to incorporate into clinical workflows, and patients may be less comfortable answering questions face-to-face about a stigmatized behavior such as substance use. – A self-administered version of the SISQs may be preferable in some practice settings, but the validity of adapting the SISQs to a self-administered format must be established before this approach can be widely recommended.
Susanne Hohler
Rwth Aachen University
Our goal was to test the sensitivity and specificity of the SISQs, as well as their feasibility, when they were self-administered by adult primary care patients using touchscreen tablet computers. Computer self-administration was chosen instead of a paper format because it may allow for easier integration with electronic health records and Web-based patient portals.
Participants and Recruitment Participants were recruited from two safety-net hospital-based adult primary care clinics; Site A was located in Boston and Site B in New York City. Data were collected at Site A in June and July 2012 and at Site B from November 2012 through June 2013. Eligible individuals were 21–65 years of age, English-speaking, and current clinic patients. We excluded individuals over age 65 because the lower prevalence of unhealthy drug and alcohol use in this age group, would not have supported meaningful analyses in our sample. Participants were recruited consecutively while they were waiting for medical appointments. At Site A, each patient who presented for a scheduled clinic visit was approached, while at Site B patients were recruited from the waiting area using a pre-specified path., The institutional review boards of New York University School of Medicine and Boston Medical Center approved all study procedures.
Stahlbau
Study Procedures Study visits were conducted in a private room, and participants were informed that responses to assessments were anonymous and confidential. Participants completed the SISQs independently using a touchscreen tablet computer. Any requests for assistance were recorded on paper by a research assistant (RA).
Following completion of the SISQs, the RA administered a series of interviews that served as reference standard comparison measures. Saliva testing was offered only to Site B participants, who were informed of the voluntary saliva test after completing all self-reported interviews, and were asked to provide a second informed consent. The total time to complete all study procedures was 30–45 min for most participants. Experimental Instruments: Single-Item Screening Questions (SISQs) for Alcohol and Drugs The computer self-administered SISQs were identical to those previously validated as interviewer-administered questions., The alcohol SISQ asked “ How many times in the past year have you had X or more drinks in a day?” ( X = 5 for men, X = 4 for women). The drug SISQ asked “ How many times in the past year have you used an illegal drug or used a prescription medication for non-medical reasons (for example, because of the experience or feeling it caused)?” Participants were instructed to enter a numeric response, and to enter zero if the answer was ‘never.’ Following the scoring system used for the interviewer-administered SISQs, responses were dichotomized, with any response greater than zero indicating a positive screen. Combination of Reference Standard Measures Defining Unhealthy Use, Current Risky Use, Problem Use, and Substance Use Disorder Unhealthy use was defined as the presence of current risky use, problem use, or a substance use disorder. Current risky use was based on response to a 30-day timeline follow-back (TLFB) interview.
An individual was classified as having risky alcohol use if s/he reported use in excess of guideline-recommended limits (5 drinks/day or 14 drinks/week for men; 4 drinks/day or 7 drinks/week for women). Risky drug use was any use of an illicit drug or misuse of a prescription medication (using more than prescribed, for reasons other than as prescribed, or without a prescription). At Site B, saliva testing provided an additional measure of current risky use. Testing was performed with the Intercept ® immunoassay (OraSure Technologies, Inc., Bethlehem, PA, USA), which has a window of detection of up to 3 days for most drugs. – To assist in the interpretation of results, participants were asked to report any medical use of medications that might be detected in the saliva test.
Problem use was defined as use in the past 12 months, with at least one self-reported consequence of use. Use in the past 12 months was assessed using the MINI-Plus (Version 6.0) screening items for alcohol and drug use. – Consequences were measured using the Short Inventory of Problems (SIP) for alcohol, and the Short Inventory of Problems for Drugs (SIP-D)., Substance use disorder was determined by the MINI-Plus (Version 6.0). – The MINI-Plus alcohol and drug modules are structured interviews to assess alcohol and drug use disorders, as defined by DSM-IV abuse or dependence. Statistical Analysis We examined descriptive statistics for the sample, including demographic characteristics and prevalence of unhealthy alcohol and drug use reported on the MINI-Plus (for past 12 months) and TLFB (for past 30 days). Based on the composite reference standard measures, we calculated the sensitivity and specificity of the SISQ-alcohol and SISQ-drug. We computed positive and negative diagnostic likelihood ratios (DLRs) as an additional measure of the diagnostic value of the screening items.
To provide a measure of discriminatory power, we computed receiver operator characteristic (ROC) curves and examined the area under each curve (AUC). Exact 95% confidence intervals (CIs) were calculated for all accuracy estimates. After completing sensitivity and specificity calculations for each site individually, we examined differences in the SISQ results (in comparison to reference standard measures) between the two sites by conducting chi-square analyses. To compare sites with respect to sensitivity, among those who were positive on the reference standard measures, we examined the cross-tabulation of site and SISQ result.
Similarly, to compare sites with respect to specificity, we examined the cross-tabulation of site and SISQ result among those who were negative on the reference standard measures. SISQ results differed significantly for the two sites only for the comparison of specificity with respect to any unhealthy drug use. At Site B, there was a higher proportion of false-positive results on the SISQ for unhealthy drug use, with a false-positive fraction (defined as false positive ÷ false positive + true negative) of 14/129 at Site B versus 5/206 at Site A; p = 0.001).
Because specificity was good at each site (98% at Site A and 89% at Site B), the decision was made to combine the two sites for all analyses. The sensitivity and specificity of the SISQs for detecting unhealthy use was then estimated for a number of pre-specified subgroups in which prior studies have found substance use screening tools to have reduced precision or feasibility., – These subgroups were as follows: male, age greater than 50, Hispanic/Latino, primary language other than English, born outside the U.S., and education or health literacy lower than high school level. To determine whether there were significant differences in SISQ accuracy for each subgroup, we performed chi-square analyses, cross-tabulating each subgrouping variable with the SISQ screening result, within groups that were positive (sensitivity) or negative (specificity) on the reference standard measures. Analyses were conducted using version 13 of Stata (StataCorp, 2013; StataCorp LP, College Station, TX, USA) and its diagnostic testing module. Sensitivity and Specificity The accuracy of the SISQs in comparison to reference standard measures is presented in Table. For both the SISQ-alcohol and the SISQ-drug, sensitivity generally increased and specificity decreased as the level of risk rose from risky consumption to problem use or substance use disorder.
Diagnostic likelihood ratios (DLRs) indicate that across both substances and all risk categories, participants with a substance use condition were at least three times as likely to have a positive screen, and were less than one-third as likely to have a negative screen. AUCs indicated good discrimination. Sensitivity, Specificity, Likelihood Ratios, and Area Under the Curve of the Single-Item Screening Questions for Alcohol and Drugs in Detecting Unhealthy Use, Current Risky Use, Problem Use and Substance Use Disorder ( N = 459) Because the SISQ-drug screens conjointly for both illicit and non-medical prescription drug use, we examined whether its sensitivity would differ among those who used prescription drugs only or illicit drugs only. Past-year illicit drug use without prescription drug use was reported by 90 participants, 64 of whom (71%) screened positive on the SISQ-drug, while past-year prescription drug use without illicit use was reported by 3 participants, 2 of whom (67%) screened positive on the SISQ-drug. Among those who participated in saliva testing for drugs, which was offered at Site B, 8 participants tested positive for at least one drug. Each of these participants also reported unhealthy drug use on the self-reported reference standard measures.
Feasibility A majority (71%) of participants were able to complete the SISQs without assistance. At Site A, most requests for assistance (27/44, 61%) were because of difficulty reading or comprehending the SISQs; 9 (29%) were for problems using the computer, and 8 (18%) were for other types of assistance. At Site B, most requests (68/88, 77%) were for problems using the computer, almost all due to confusion about how to advance to the next question. At Site B, 10 (11%) participants had problems reading or comprehending the questions, and 10 (11%) requested other types of assistance. The SISQs had higher sensitivity and lower specificity for detecting unhealthy use among participants who did not request assistance.
With respect to alcohol, sensitivity was 77.5% (95% CI 68.6–84.9) and specificity was 81.9% (95% CI 76.2–86.8) in those who did not request assistance, while sensitivity was 60.0% (95% CI 42.1–76.1) and specificity was 90.7% (95% CI 83.1–95.7) in those who did. For drugs, a similar pattern was observed: sensitivity was 76.3% (95% CI 66.4–84.5) and specificity was 93.6% (95% CI 89.6–96.4) in those who did not request assistance, and sensitivity was 55.2% (95% CI 35.7–73.6) and specificity 96.1% (95% CI 90.3–98.9) in those who did. DISCUSSION The self-administered alcohol and drug SISQs each had adequate sensitivity and high specificity for the detection of unhealthy substance use in this sample of adult primary care patients.
While the self-administered SISQs did not have the very high sensitivity and specificity observed for the interviewer-administered versions, the modest decrement in accuracy must be weighed against the potential advantages of a self-administered approach. These advantages include the ability to accomplish screening prior to the medical encounter, maintaining fidelity even when used outside of a tightly controlled research setting, and the more open disclosure of stigmatized behaviors that is achieved with self-administered questionnaires., – The differences in performance of the self-administered versus interviewer-administered SISQs could also be driven by differences in study populations. The interviewer-administered SISQs were validated in a single site, while the current study drew from two geographically distinct primary care clinics, serving populations with varied demographic and substance use characteristics. All brief substance use screening tools have accuracy limitations, and their performance can vary depending on the population and the context in which they are administered.
The SISQs have similar sensitivity and specificity to those of other commonly used screening tools, such as the Alcohol Use Disorders Identification Test consumption items (AUDIT-C), and the Drug Abuse Screening Test (DAST-10), both of which are significantly longer and relied on a trained interviewer to deliver them in the validation studies. The accuracy of computer self-administered SISQs might be improved by modifying their delivery to improve comprehension and usability. Many participants requested some assistance in completing the SISQs, and sensitivity was substantially lower in these individuals. Still others may have had difficulty understanding the questions, but did not ask for assistance.
It is possible that modifying the self-administered SISQs by providing structured response categories, simplifying the language, and providing definitions of terms, or by using an audio computer-assisted self-interview (ACASI) approach to accommodate those with lower literacy, could improve their accuracy. Most of the assistance requests were from Site B, and almost all were due to participants having to touch a “Next question” button to advance to the next item. At Site A, a brief tutorial that participants viewed on the computer prior to answering the SISQs likely mitigated this problem. While this type of technical issue could pose a threat to the feasibility of integrating computer self-administered screening in routine care, it may be avoided through simple modifications of the user interface and usability testing prior to implementation. Brief screening tools like the self-administered SISQs may need to be followed by further assessment to guide clinical interventions.
Assessment would be required to identify what substances (other than alcohol) a patient was using, and may be needed to distinguish between individuals with risky or problem use versus dependence. While the interviewer-administered SISQs appear to discriminate between unhealthy use and dependence with good precision by applying different cutoffs, further analysis of self-administered SISQs is needed to determine whether these cutoffs can be similarly applied. Limitations Our study does have limitations. Both samples were drawn from East Coast safety-net hospital-based primary care clinics, and thus our findings may not fully generalize to other populations.
We do not know, for example, how the self-administered SISQs would perform in non-urban settings or in populations with a lower prevalence of illicit drug use. While we would anticipate comprehension of the SISQs to be easier for primary care populations with higher levels of formal education and fewer non-native English speakers, there could be unforeseen problems with their feasibility or acceptability. Similarly, individuals over the age of 65, who may differ in their responses to computer self-administered screening tools or may find them more challenging to operate, were excluded from our study.
Many patients were ineligible for the study due to lack of English fluency, indicating that having SISQs available in other languages would be important for their adoption in clinical practice. We tested only computer self-administered versions of the SISQs, and cannot anticipate how results might differ if they were administered on paper. However, the format of the questions was identical to what would be given in a paper-based version. A limitation of our study design was the lack of biological measures at Site A and the use of only saliva testing at Site B.
However, the accuracy of the SISQs was supported by the biological measures that were collected, which detected no cases of drug misuse that were not already reported on the self-reported reference standard instruments. Perhaps the most important limitation of our findings is that the SISQs were administered anonymously and with an assurance of confidentiality. We do not know how our findings would have been influenced had participants been informed that their medical provider or other clinic staff would receive their results. This is an important area of future research for screening tools like the SISQs, which are intended for use in clinical practice.