Background: A common criticism of systematic reviews (SRs) is that they carry out statistical pooling without adequately considering clinical heterogeneity. Findings based on such estimates could be misleading as readers often consider the pooled summary estimates and ignore any underlying heterogeneity.

Objectives: To analyse the systematic reviews in KSR Evidence (a database of systematic reviews) which were assessed to be at high risk of bias (RoB) because pooled analysis was not appropriate due to presence of clinical heterogeneity.

Methods: All SRs in the database are critically appraised using the validated Risk of Bias in Systematic Reviews (ROBIS) tool, specifically designed to assess RoB in SRs and are checked for quality by a second independent reviewer. We analysed a sample of appraisals independently checked by 10 experienced reviewers. We identified reviews judged at high RoB for the synthesis as the authors pooled the studies despite of the clinical heterogeneity across the studies.

Results: The database has >30,000 SRs, as identified by our searches to date. This analysis includes a sample of 651 SR critical appraisals (checked independently) of which 49% (321) were judged at high RoB on ROBIS domain 4 (Synthesis and Findings). Pooling was judged to be inappropriate, due to the presence of clinical heterogeneity, in 17.5% (56/321) of the SRs judged to be at high RoB. Sources of heterogeneity across the 56 SRs included pooling of different study designs, time points, scales used for measuring outcomes and interventions/comparators with different dose regimens, etc. The majority of these SRs (45/56) also showed statistical evidence of heterogeneity.

Conclusions: Consideration of clinical heterogeneity before pooling studies is crucial. A large number of studies ignore clinical heterogeneity and report summary estimates which may mask clinically important variations in effectiveness. SRs should not conduct pooled analysis where inappropriate and further improve reporting of synthesis.