Not all positive outcomes are created equal!

Much of the crucial work in clinical trials occurs before they begin. Statisticians first identify the primary outcome they are focusing on and make numerous assumptions about its frequency based on previous studies. They also estimate the likely difference in outcomes between the groups due to the intervention being tested.

Sometimes their estimates are accurate, and other times they are not. For primary outcomes, the medical field has agreed to accept a 5% chance of a type I error (false positive / false alarm) and a 20% chance of a type II error (false negative / missing the boar). We can explore this further when discussing frequentist versus Bayesian analysis and publication bias.

Researchers also identify other potentially interesting outcomes to explore, though these are not as critical as the primary outcome. These secondary outcomes are used to generate hypotheses and do not require the same assumptions about the number of events.

It is essential to register the trial in advance, clearly defining the primary outcome. This helps prevent the temptation to retrospectively change a positive secondary outcome into a primary outcome, a practice that has been noted in the past and can be very misleading and prone to bias.

The main reason for not using numerous outcomes is the limitation of statistical methods. The more outcomes you examine, the greater the likelihood of finding a difference between groups (a positive result) purely by chance. If each outcome has a 5% chance of being a false positive, examining 20 outcomes almost guarantees a false positive. The same principle applies to subgroup analysis.

We are all susceptible to biases and the tendency to interpret results to confirm our existing beliefs (confirmation bias). I have personally experienced this, and the only remedy is being aware of our cognitive biases. Understanding these concepts provides a framework to avoid repeating past mistakes. Misunderstandings of the scientific method can lead to mistrust in the medical field. This gap may be exploited by unscrupulous individuals who do not base their claims on sound science, potentially causing harm through ineffective advice or delayed care.

1. Vetter TR, Mascha EJ. Defining the Primary Outcomes and Justifying Secondary Outcomes of a Study: Usually, the Fewer, the Better. Anesth Analg. 2017 Aug;125(2):678-681. doi: 10.1213/ANE.0000000000002224. PMID: 28682958.

2. Jaiswal J, Halkitis PN. Towards a More Inclusive and Dynamic Understanding of Medical Mistrust Informed by Science. Behav Med. 2019 Apr-Jun;45(2):79-85. doi: 10.1080/08964289.2019.1619511. PMID: 31343962; PMCID: PMC7808310.

3. Ricotta EE, Bents S, Lawler B, Smith BA, Majumder MS. Search interest in alleged COVID-19 treatments over the pandemic period: the impact of mass news media. medRxiv [Preprint]. 2024 Nov 22:2024.11.20.24317650. doi: 10.1101/2024.11.20.24317650. PMID: 39606346; PMCID: PMC11601752.