ATOP Validation Studies
The COQI Program has conducted a number of studies confirming that the ATOP is valid and reliable to use as a clinical and research tool by Drug and Alcohol Treatment providers.
Assessing the concurrent validity, inter‐rater reliability and test‐retest reliability of the Australian Treatment Outcomes Profile (ATOP) in alcohol and opioid treatment populations
https://doi.org/10.1111/add.15331
Background and Aims
The Australian Treatment Outcomes Profile (ATOP) is a brief instrument measuring recent substance use, risk profile, and general health and wellbeing among clients attending Alcohol and other Drug (AoD) treatment services. This study evaluates the ATOP for concurrent validity, inter‐rater and test‐retest reliability among alcohol and opioid treatment groups.
Design
For concurrent validity and inter‐rater reliability, participants completed an ATOP with a clinician, and an ATOP plus standardised questionnaires (Time Line‐Follow Back, Opiate Treatment Index, Kessler‐10, 12 item Short Form Survey, WHO Quality of Life‐BREF, Personal Wellbeing Index) with a researcher within 3 days. For test‐retest reliability, participants completed two ATOPs with a researcher within a 3‐day interval.
Setting
Outpatient AoD treatment centres in Australia.
Participants
For testing concurrent validity and inter‐rater reliability, 278 participants were recruited by advertisements in waiting rooms or clinician invitation over 2016‐2018. A further 94 participants were recruited to examine test‐retest reliability
Measurements
Statistical tests used for concurrent validity and test‐retest reliability were Pearson’s and Spearman rank order correlations for continuous variables, and Cohen’s κ for nominal variables. Inter‐rater reliability was assessed using Krippendorf’s α.
Findings
Most Australian Treatment Outcomes Profile items returned excellent or moderate validity and reliability. For the main substances used – alcohol, cannabis, and benzodiazepines – concurrent validity, interrater reliability and test retest reliability all reached excellent or good agreement (0.72‐0.98). Psychological health, physical health and quality of life showed fair to strong agreement with their comparator scales (0.47‐0.85).
Conclusions
The Australian Treatment Outcomes Profile is a validated and reliable instrument for assessing recent substance use and clinical risk, health and welfare among alcohol and opioid clients in Alcohol and other Drug treatment settings. Its ability to reliably measure complex constructs, such as psychological and physical health, against longer scales makes it suitable for integration into routine clinical care, enabling regular monitoring of patient outcomes and safety parameters.
Validation and implementation of the Australian Treatment Outcomes Profile in specialist drug and alcohol settings
https://doi.org/10.1111/dar.12083
Introduction and Aims
The measurement of clinical outcomes is an important, but lacking, component of drug and alcohol treatment in Australia. This study aimed to psychometrically validate the Treatment Outcomes Profile under Australian conditions, examining implementation and feasibility issues in three public opioid treatment program clinics in NSW.
Design and Methods: The Treatment Outcomes Profile was modified to reflect Australian conditions and re‐named the Australian Treatment Outcomes Profile (ATOP). The ATOP was introduced into the participating clinics and administered by clinic staff at 3‐month intervals as part of routine clinical practice. Participants completed a research interview, consisting of the ATOP and a suite of ‘gold standard’ instruments assessing substance use and related health and welfare domains, in the 72 h following completion of a routine clinical ATOP. The researcher‐ and clinician‐administered ATOPs were compared to assess interrater reliability, and the researcher‐administered ATOP and ‘gold standard’ instruments were compared to assess concurrent validity. Implementation and feasibility issues were assessed using questionnaires and focus groups with clinician and clients.
Results: The ATOP demonstrated acceptable concurrent validity and interrater reliability. It was well received by clients and clinicians, particularly for its ease of use, applicability and brevity.
Conclusions: The ATOP is a psychometrically valid instrument for the measurement of treatment outcomes in Australian opioid treatment populations and can feasibly be implemented as part of routine clinical practice in specialist opioid treatment program clinics. The role of the ATOP to measure outcomes in other drug and alcohol treatment modalities requires exploration.
Assessing the validity of the Australian Treatment Outcomes Profile for telephone administration in drug health treatment populations
https://doi.org/10.1111/dar.13088
Introduction and Aims
The Australian Treatment Outcomes Profile (ATOP) is a brief clinical tool measuring recent substance use, health and wellbeing among clients attending alcohol and other drug (AOD) treatment services. It has previously been assessed for concurrent validity and inter‐rater reliability. In this study we examine whether it is suitable for administration over the telephone.
Design and Methods
We recruited a sample of 107 AOD clients across public sector specialist AOD treatment services in New South Wales, Australia between 2016 and 2018. Participants had a mean age of 47 years and 46% were female. Participants completed a face‐to‐face ATOP and a phone ATOP with a researcher within 5 days. Comparisons between the two administration modes were undertaken using Spearman's rank correlation coefficient for continuous or ordinal variables, and Cohen's Kappa for nominal variables.
Results
Among 107 participants, 59% were attending for alcohol treatment and 41% for opioid treatment. Most ATOP items (76%) reached above 0.7 (good) or 0.9 (excellent) agreement between face‐to‐face and telephone use.
Discussion and Conclusions
Our findings suggest that the ATOP is a suitable instrument for telephone monitoring of recent substance use, health and social functioning among AOD clients. Its validation for remote use over the telephone will support staff to monitor clients' risks and outcomes—of particular relevance in response to the COVID‐19 pandemic in which services are increasingly relying on telework approaches to client monitoring.
Validation of the Australian Treatment Outcomes Profile for use in clients with cannabis dependence
https://doi.org/10.1111/dar.13050
Background
Over one fifth (22%) of all clients seeking treatment for drug and alcohol dependence list cannabis as their main drug of concern. However these clients are often quite different in their patterns of substance use and often have fewer social, mental health, and physical health issues. It is important we ensure that the ATOP is valid and reliable for use in this large group of clients.
Methods
We analysed data from 128 heavily dependent cannabis users attending alcohol and other drug treatment services in South Eastern Sydney Local Health District. ATOP Psychological Health, Physical Health, and Quality of Life questions had their validity tested by comparing them to answers on already-validated scales that measured the same thing. All ATOP questions also had their reliability tested by administering the ATOP twice with two different raters.
Results
ATOP Psychological Health showed moderate to strong agreement with SF-36 Mental Components, SF-36 Mental Health and DASS-21 scores (r = 0.40–0.52) and ATOP Physical Health with SF-36 Physical Components and SF-36 General Health scores (r = 0.36–0.67). The ATOP Quality of Life scale showed moderate agreement with the Sheehan Disability Scale and SF-6D (r = 0.38–0.40). ATOP substance use, employment, education and child care items showed good to excellent interrater reliability (Krippendorff’s α = 0.62–0.81), and tobacco use, Psychological Health, Physical Health and Quality of Life showed fair to moderate interrater reliability (Krippendorff’s α = 0.42–0.53)
Conclusion
The ATOP appears to have good validity and reliability when used with clients who are seeking treatment for cannabis dependence, and is suitable for widespread use in this population.
The Australian Treatment Outcomes Profile instrument as a clinical tool for older alcohol and other drug clients: A validation study
https://doi.org/10.1111/dar.12393
Background
There is evidence that long-term substance use speeds up the harmful effects of aging on cognitive function, psychological and physical health. As a result older drug and alcohol clients have different needs to younger clients. About 13% of all drug and alcohol treatment episodes in Australia were for people 50 years or older. It is important that the ATOP be validated for use in this group of clients.
Methods
The validity of the ATOP was tested using 99 clients attending alcohol and other drug treatment services in South Eastern Sydney Local Health District. ATOP questions were compared to answers on already-validated scales that measured the same thing.
Results
The number of days clients reported they had used alcohol in the previous 28 days on the ATOP had excellent agreement (r = 0.75) with scores on the Alcohol Use Disorders Identification Test. ATOP Psychological Health and ATOP Quality of Life questions had strong agreement (r = -0.65 and r = -0.61 respectively) with the Geriatric Depression Scale 10. The ATOP Physical Health question had moderate agreement (r = 0.53) with the SF-12 Physical Health scale.
Conclusions
The moderate to strong agreement of the ATOP questions with validated scales showed the ATOP is valid for use in older clients with alcohol or other drug dependence.
Determining clinical cutoff scores for the Australian Treatment Outcomes Profile psychological health, physical health and quality of life questions.
https://doi.org/10.1111/dar.13346
Introduction
The Australian Treatment Outcomes Profile (ATOP) is a brief instrument that measures self-reported substance use, health, and wellbeing in the previous 28 days for people in alcohol and other drug treatment. Previous studies have established the concurrent validity, inter-rater, and test–retest reliability of the tool. The current study sought to identify recommended cutoff scores for ATOP items for psychological health, physical health and quality of life that identify clients reporting clinically significant problems warranting further assessment and/or intervention, compared to cutoffs used by ‘gold standard’ measures for these domains.
Methods
Clients attending for treatment for problems with opioid (n = 144) or alcohol use (n = 134) completed the ATOP and comparison standardised questionnaires (Kessler-10, Short Form Survey 12 and the Personal Wellbeing Index) with a researcher. Receiver operating characteristics analysis, along with clinician perspectives, were used to recommend cutoff scores for ATOP items indicative of clinically significant problems.
Results
A cutoff score of 5 or less out of 10 was identified as an optimal pragmatic cutoff for ATOP items relating to psychological health, physical health and quality of life items with regards to balancing sensitivity, specificity, and application in a treatment setting.
Discussion and Conclusions.
The recommended clinical cutoffs will support clinicians and treatment services to identify clients who require further assessment and follow up for their psychological health, physical health and quality of life. The current study provides further evidence for the utility of the ATOP for individual clinical review, service planning and research.