Background: Debates regarding the credibility of effect modifiers are often contentious. Although it is desirable to identify effect modifiers that explain heterogeneity of treatment effects, subgroup analyses may lead to spurious inferences of subgroup effects in randomised trials and meta-analysis. Authorities have, in response, suggested varying criteria to assess the credibility of effect modifiers. A formal, consensus-based instrument remains unavailable.

Objectives: To develop an instrument to assess the credibility of putative effect modifiers in randomised trials and meta-analyses.

Methods: We will follow a rigorous instrument-development process, which will involve expert panels and users. First, we will perform a qualitative systematic survey of the methodological literature discussing credibility of effect modifiers. We systematically searched MEDLINE, EMBASE and Textbooks and identified 409 potentially relevant full texts. These reports will serve as the basis for identifying experts in subgroup analysis and generating candidate items for the new instrument. We are currently abstracting reported credibility criteria (e.g. pre-specification, test of interaction, small number of subgroup analyses), rationales, and context (e.g. in trials or meta-analyses, purpose of subgroup analysis) using both original quotes and a newly developed taxonomy. We will randomly choose 20 experts who will form two panels. Panel 1 will be involved in the instrument development and panel 2 in the testing phase. In addition, we will involve two groups of 20 users who will apply the draft instrument to a sample of subgroup analyses using formal user testing methods. We will test the final instrument in a reliability study.

Discussion: At summit, we will present the concept, the item selection process, and the draft instrument. The new instrument will have immediate impact on the analysis, interpretation, and reporting of effect modifiers in individual trials and meta-analyses.