Background: We modified the risk-of-bias (RoB) tool for non-randomised studies of interventions (ROBINS-I) for use in studies of environmental and occupational exposures (ROBINS for exposures). Objectives: To assess reliability and validity of ROBINS for exposures through comparison with other tools, external evaluation and integration of results from application of the instrument into the GRADE framework for evidence assessment.
Methods: Two raters independently applied ROBINS for exposures to 7 systematic reviews assessing the impact of environmental exposures on health outcomes. Topic-specific experts reviewed study-level RoB judgments and rationale for accuracy. We determined RoB across the body of evidence for each outcome, integrating that judgment into a GRADE evidence assessment. To determine reliability, 3 raters applied ROBINS for exposures and 3 commonly used RoB instruments for environmental exposure studies (Newcastle-Ottawa Scale, and tools used by the National Toxicology Programs’ Office of Health Assessment and Translation, and Office of the Report of Carcinogens) to a subset of 5 or 6 primary studies within 5 of the systematic reviews. To measure external validity, PhD-level exposure topic-specific experts provided 160 unstructured RoB assessments of the same subset of studies.
Results: Assessment of the 7 systematic reviews did not identify any individual study or body of evidence judged as 'Low' RoB (equivalent to a well-conducted randomised trial). Assessments across the body of evidence for different outcomes demonstrated examples of 'Moderate', 'Serious', and 'Critical' RoB. Within GRADE, these translated to at least 'Very Serious' RoB and 'Low' certainty in the evidence. We did not identify any examples for which the body of evidence would not be rated down. We will present reliability and validity analyses. Completion of individual study assessments varied from 10 to 60 minutes depending on instrument and study.
Conclusions: Examples from the application of ROBINS for exposures to environmental studies can inform and guide systematic review and guideline developers, increasing the transparency and rigour of the evidence assessment.