Stopping trials early

Stopping trials early in an epidemic in research. Truncated RCTs (tRCTs) tend to overestimate effects and cause a lot of downstream effects. Overestimated effects cause a lot of attention and tend to change practice standards (and then cause backlash when they aren't reproduced), they decrease the interest in conducting more trials in the same topic and they will sometimes exert too much influence in systematic reviews and meta-analysis, overall over-estimating our understanding of the actual effect of drugs. The overestimation caused by truncating the RCTs is independent of all the other validity criteria.

We have been able to study it by comparing tRCTs to completed RCTs and because investigators continued studies despite meeting criteria for stopping, thankfully due to realizing the results were too surprising or too good to be true.

When compared to completed studies with the same intervention, truncated RCTs show overestimation of results by about 20%, whether their results were positive, no difference or negative (in the last cases, showing false positive studies).

Researchers have also continued trials despite having met a rule for stopping early and showed that the results ended up being more moderate at the end of the study (but still likely a little over-estimated). Getting closer to the truth in this case is due to regression to the mean. If too many patients in one group achieved an event by random chance initially, if we wait long enough, the other group will accrue more events too. The higher number of events, will give us a result closer to the truth.

If the intervention effect is too good to be true, it probably is. It is important to remember that disease processes are multifactorial and any intervention will likely only have a small effect at best. True large effects are uncommon.

Smaller samples sizes / event rates from stopping to early will cause a larger overestimation.

The same concepts apply to stopping trials for no benefit and for harm.

What about ethical concerns? The problem lies in potentially depriving a small group of patients of a benefitial intervention, however, we could be causing harm or using unhelpful interventions in large populations as a result. The harm to the population as a whole outweighs the potential harm to the individuals in the study.

How about using pre-established stopping rules? It doesn't seem to protect against overestimation of effects, as this trials that were stopped early have already an overestimation due to random chance alone. Even changing the p value doesn't seem to help (it makes it less likely to stop early, but it doesn't help when the boundary is crossed).

How about stopping early for inability to recruit enough patients? In this case there seems to be no overestimation. However, not having reached enough patients / events as required in the power calculation done beforehand increased the risk of a false negative study - not seeing a true difference in the study.

When can we believe results from a tRCT? Tough question! Always use caution.

But they are less likely to mislead us if:

• There are large numbers of events.

• They do not use composite outcomes in the stopping rule.

• They show similar results to their non truncated counterparts.

• They stopped after almost completing the trial, as opposed to early on.

References:

-Guyatt, G, et al. (2015). JAMA Users guide to the medical literature. McGraw-Hill.

-Briel M, Bassler D, Wang AT, Guyatt GH, Montori VM. The dangers of stopping a trial too early. Journal of Bone and Joint Surgery. 2012;94(Supplement_1):56-60.

-Guyatt GH, Briel M, Glasziou P, Bassler D, Montori VM. Problems of stopping trials early. BMJ. 2012;344(jun15 1):e3863-e3863.