PICO annotation: harnessing crowds and experts in making health evidence more discoverable and re-usable

ID: 

18950

Session: 

Long oral session 19: Linked data & data sharing

Date: 

Friday 15 September 2017 - 11:00 to 12:30

Location: 

All authors in correct order:

Beecher D1, Mavergames C2, Becker L3, Wisniewski S4, Noel-Storr A5
1 CET/IKMD Cochrane, Italy
2 CET/IKMD Cochrane, Germany
3 Cochrane, USA
4 Cochrane Crowd, UK
5 Cochrane Dementia and Cognitive Improvement/Cochrane Crowd, UK
Presenting author and contact person

Presenting author:

Deirdre Beecher

Contact person:

Abstract text
Background: As part of the Cochrane Linked data project, we are annotating all Cochrane systematic reviews. Phase I, which ends in March 2017, involves annotating child-health reviews. The reviews are being annotated at two levels: review level (i.e. the question) and included study level (generating a PICO for each study). This work is part of a project funded by the Bill and Melinda Gates Foundation to generate a PICO-annotated evidence base of all Pregnancy and Neonatal Cochrane Reviews.
Similarly, Cochrane Crowd (http://crowd.cochrane.org) has now rolled out a beta PICO task. This crowd task is aimed at surfacing the core PICO elements from reports of randomised controlled trials in the areas of pregnancy and child health. The outputs of each will feed the PICO concept browser and PICO Finder: two user interfaces that facilitate discovery of Cochrane Reviews and their included trials according to PICO.

Objectives: 1. To annotate all reviews relevant to child health using the Linked data annotator tool and ontology developed by the Cochrane Linked data team.
2. To create a Crowd PICO task that would enable high-quality annotation of titles and abstracts by a Crowd.

Methods: For the Crowd annotation, a search was run to identify potentially relevant reports of trials that had not yet been included in Cochrane reviews. We fed these citations into the beta Crowd PICO task. We developed a training module to help guide Crowd contributors through an interactive example. We then made the task available to all those who had screened a certain number of records for the RCT task.
For the annotation of reviews, we focused our efforts on those reviews from the Neonatal and Pregnancy review groups. We recruited a small team of annotators and worked closely with Cochrane information specialists to ensure quality and consistent annotations.

Results: We will present results on the number of annotations performed by the Crowd and Expert annotators and the quality of those annotations across the four PICO elements.