Assessing and validating search strategies: When is a study truly identified?

ID: 

2043

Session: 

Poster session 2 Thursday: Evidence synthesis - methods / improving conduct and reporting

Date: 

Thursday 14 September 2017 - 12:30 to 14:00

Location: 

Exhibition Halls 1 & 2

All authors in correct order:

Klerings I1, Nussbaumer-Streit B1, van Noord M2, Mittermayer T3, Heise T4, Stratil J5, Lhachimi S4, Wagner G1, Dobrescu A6, Armijo Olivo S7, Teufer B1, Gartlehner G1
1 Cochrane Austria, Austria
2 Medical Center Library & Archives, School of Medicine, Duke University, USA
3 Ludwig Boltzmann Institution for Health Technology Assessment, Austria
4 Leibniz Institute for Prevention Research and Epidemiology - BIPS, Germany
5 Institute for Medical Informatics, Biometry and Epidemiology, Ludwig-Maximilians-University, Munich, Germany
6 Genetics Department, Victor Babes University of Medicine and Pharmacy Timisoara, Romania
7 Faculty of Rehabilitation Medicine, University of Alberta & Institute of Health Economics, Edmonton, Alberta, Canada
Presenting author and contact person

Presenting author:

Irma Klerings

Contact person:

Irma Klerings
Abstract text
Background: Systematic reviews employ comprehensive search strategies to identify all relevant studies on a topic. Methods projects assessing novel search approaches often benchmark the sensitivity of searches by checking whether they find the same studies as comprehensive, systematic searches. However, results of studies are frequently reported in multiple publications. Defining the threshold that classifies a study as 'identified' by a search partially determines the validity of the search strategy under assessment: The least time-intensive approach is to check if the major publication of that study is found. A second approach is to check if any publication of a study was identified. The strictest threshold requires every publication belonging to a study to be found.

Objectives: Our aim is to test the validity of three methods of classifying a study as found by a search compared to a gold standard.

Methods: As part of a larger methods project (1) we randomly chose 60 Cochrane Reviews on various clinical topics. We reproduced their MEDLINE, CENTRAL, and EMBASE searches, and used a 'content-based' approach to determine if a study was found: We read all identified publications belonging to this study and checked if they reported the outcomes relevant for the main meta-analyses. This is our gold standard method: It allows us to define a study as identified when all relevant data are found. For this study we will compare this exhaustive process with the three pragmatic approaches: a study is considered as 'identified' if a) the main study publication is found, b) one publication is found or c) all publications belonging to one study are found. We will assess to what degree these three methods agree with the results of the content-based approach.
Results will be available at the Summit.

Conclusions: To our knowledge, this is the first study to inform methodologists what is the most valid approach to define a study as found by a search strategy.

(1) Nussbaumer-Streit B, et.al. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Sys rev. 2016 Nov 22;5(1):197.