Development and validity testing of the AGREE-HS, a health systems guidance quality-appraisal tool

ID: 

19207

Session: 

Long oral session 6: Guideline development

Date: 

Wednesday 13 September 2017 - 16:00 to 17:30

Location: 

Meeting Room 1.41

All authors in correct order:

Brouwers M1, Lavis J1, Vukmirovic M1, Spithoff K1, Ako-Arrey D1, Florez ID1
1 McMaster University, Canada
Presenting author and contact person

Presenting author:

Melissa Brouwers

Contact person:

Melissa Brouwers
Karen Spithoff
Abstract text
Background: Health systems guidance (HSG) documents contain systematically developed statements or recommendations intended to address a health-system challenge. Health-system challenges are most often associated with health-services delivery, governance structures and/or financial arrangements. While numerous tools exist related to clinical practice guidelines, no tools are available to guide the development, reporting and appraisal of HSG.

Objectives: The purpose of this study was to develop a tool to assist with the development, reporting and appraisal of HSG and to test the validity of the tool.

Methods:The draft AGREE-Health Systems (AGREE-HS) tool was developed based on a critical interpretive synthesis of the literature and formal consultation with health-systems experts. To assess the face validity of the AGREE-HS, health-systems researchers, administrators and policy makers were invited to review it and provide feedback about its content and structure by completing an online survey. The survey included both Likert scale and open-ended questions.

Results: The draft AGREE-HS consists of 5 items that are individually scored on a 7-point response scale. Thirty individuals, representing all 6 WHO geographical regions, reviewed the AGREE-HS and completed the survey. Overall, respondents indicated that the AGREE-HS would be useful for guiding HSG quality appraisal (90%), HSG development (73.3%) and directing what to report in a HSG document (70%). Respondents agreed that they felt confident in applying the tool, based on the instructions provided (mean 5.7/7). Additionally, respondents agreed that the structure of the tool was logical and comprehensive. Key qualitative feedback suggested clarifying the interpretation of items and overall scores.

Conclusions:The results of this validity study support the use of the AGREE-HS in the development, reporting and appraisal of HSG. Survey results were used to further refine the tool. The refined tool is being applied in a follow-up study to appraise the quality of current HSG documents; this status report will serve as a baseline upon which to measure future improvement in the quality of HSG.