Background: We illustrated an approach of teaching randomised-controlled trials through a 'Kit Kat' trial with participants at the 24th Cochrane Colloquium in Seoul, Korea. The trial ran in four cohorts of tasters where three out of four Kit Kats from the United States (US), United Kingdom (UK), France, and South Africa were compared in each cohort.

Objective: To illustrate the challenges in analysing and interpreting rank statistics in the context of a network of trials

Methods: We used a crossover design. Each taster tried a piece of three different Kit Kats, wrapped in foil, in a random order. We concealed the random sequence using opaque stickers that were peeled off at time of randomisation. Participants indicated which Kit Kat they liked the 'most', 'second most', and 'least'. This design created a network of four 'nodes' for comparison where the outcome of interest is rank statistics.

We summarised the ranks by cohort. We examined whether there is evidence for carry-over effect and cohort effect. We pooled the differences of proportion of each rank in a fixed effect meta-analysis. We also fit Bradley-Terry models to estimate the latent 'likeness' of each Kit Kat by pooling ranks across cohorts while acknowledging the cohort effects. Such latent likeness can be used to rank the preference of Kit Kats, or predict the outcome of a comparison, e.g. probability of one Kit Kat is preferred over another.

Results: 126 conference attendees from 33 different countries participated from 24 - 26 October 2016. We observed a cohort effect, which may explain inconsistent rankings between cohorts (Figure). The pairwise meta-analyses suggest that Kit Kats from France were never 'liked the most' and Kit Kats from the UK were never 'liked the least'.

Bradley-Terry model suggested that Kit Kats from South Africa (likeness value= 0.35; the higher the more likable) are liked more than Kit Kats from the UK (0.22), US (0.14), or France (reference= 0) (Table).

Conclusions: Meta-analysis may be an appropriate method to combine rank statistics. The statisticians we consulted also suggested different approaches, which we will present at the Summit.