Systematic reviews assessed as high risk of bias due to avoidable failures in searching: analysis of a data set of critically appraised systematic reviews

ID: 

18356

Session: 

Long oral session 21: Issues in systematic review methods

Date: 

Saturday 16 September 2017 - 11:00 to 12:30

Location: 

All authors in correct order:

de Kock S1, Stirk L1, Noake C1, Deshpande S1, Misso K1, Kleijnen J1, Duffy S1
1 Kleijnen Systematic Reviews Ltd, United Kingdom
Presenting author and contact person

Presenting author:

Shelley De Kock

Contact person:

Abstract text
Background: A substantial number of systematic reviews (SR) are failing to follow recommendations for the conduct and reporting of search methods despite the availability of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and the Cochrane Handbook for Systematic Reviews of Interventions.

Objectives: We aimed to analyse the main reasons for SRs being assessed as high risk of bias in Domain 2 (Identification and Selection of Studies) of the ROBIS Risk-of-Bias tool.

Methods: The KSR Evidence database has over 30 000 SRs critically appraised according to the ROBIS tool. These were filtered to those assessed to be of high risk of bias in Domain 2 and, from this set, a random sample was selected for further analysis.

Results: From our piloted sample we found the most frequent reasons for SRs receiving a high risk of bias assessment in Domain 2 was the failure to search for non-English publications; and no undertaking of supplementary searches additional to healthcare database searching, i.e. no hand searching, grey literature or citation searching. A substantial number of SRs failed to report search strategies and/or the study selection process adequately. Of the SRs assessed as high risk of bias in Domain 2, 98 per cent are ultimately assessed as having a high risk of bias overall for the full review.

Conclusion: SRs are being assessed as of high risk of bias in Domain 2 for reasons which, in some cases, could easily be avoided. Improved reporting and omission of language limits would add little extra work but could improve the academic rigour of and increase the value of the research undertaken. As SRs are expected to be transparent and reproducible, we believe common failures in this domain undermine the overall value of SRs and, in so doing, contribute to unnecessary research waste.