When should systematic reviews be replicated, and when is it wasteful? An analysis of reasons for discordance among overlapping systematic reviews

ID: 

19213

Session: 

Short oral session 6: Evidence synthesis methods

Date: 

Thursday 14 September 2017 - 11:00 to 12:30

Location: 

Meeting Room 1.63

All authors in correct order:

Karunananthan S1, Welch V2, Grimshaw J1, Maxwell L3, Avey M4, Batista R3, Curran J5, Ghogomu E2, Graham I3, Ioannidis J6, Jordan Z7, Jull J2, Lyddiatt A8, Moher D1, Ngobi JB3, Pardo J1, Petkovic J9, Petticrew M10, Pottie K2, Rada G11, Rader T12, Shamseer L1, Shea B1, Siontis K13, Smith C3, Tschirhart N14, Vachon B15, Wells G3, White H16, Tugwell P3
1 Ottawa Hospital Research Institute, Canada
2 Bruyère Research Institute, Canada
3 University of Ottawa, Canada
4 National Institutes of Health, United States
5 Dalhousie University, Canada
6 Stanford University, United States
7 University of Adelaide, Australia
8 Cochrane Consumer Network, Canada
9 University of Split, Croatia
10 London School of Hygiene & Tropical Medicine, United Kingdom
11 Universidad Católica de Chile, Chile
12 Freelance Information Specialist, Canada
13 University of Michigan, United States
14 University of Oslo, Norway
15 Université de Montréal, Canada
16 Campbell Collaboration, Norway
Presenting author and contact person

Presenting author:

Vivian Welch

Contact person:

Sathya Karunananthan
Vivian Welch
Peter Tugwell
Abstract text
Background: Replication is a cornerstone of the scientific method. However, unnecessary duplication rather than replication is unethical and a cause of research waste. Moreover, what appear to be duplicate systematic reviews (SRs) often come to different conclusions. Multiple overlapping SRs with not infrequent discordance lead to confusion among users (e.g. patients, healthcare workers, social workers). A better understanding of the reasons for discord among overlapping SRs may contribute to the development of guidance on when to replicate a SR, and when not to.

Objectives: To develop a checklist to identify reasons for discordance among overlapping SRs.

Methods: Based on a review of the literature and consultation with experts, we developed a checklist of items to understand reasons for discord among overlapping SRs. We tested the feasibility and usefulness of the checklist on several overlapping SRs with discordant results or conclusions.

Results: The checklist itemises components of the objectives, methods for study inclusion, selection of outcomes, data synthesis, reporting and interpretation of findings, which may contribute to discordant findings in overlapping SRs. Information on author discipline and affiliation, conflict of interest, and SR quality was also recorded. The checklist was tested on a diverse selection of discordant reviews in controversial areas including deworming, glucosamine, vitamin D supplementation, payment for environmental services, and pre-school programmes. The most frequent reasons for discord included differences in study eligibility criteria and definition of outcomes, leading to differences in the primary studies being reviewed. We noted several examples where review conclusions supported possible bias related to reviewer conflict of interest.

Conclusions: The checklist for discordant SRs is a useful tool for explaining discordant findings among overlapping SRs. Development of this tool is part of a larger project to establish guidance on when replication of SRs may be useful, and when it would be wasteful. This work aims to support reliance on high-quality SRs rather than low-quality duplication.