Are my results robust or fragile?

One of the ways of assessing if a study is fragile i.e. random chance could have made it significant, is to look at the fragility index (FI). The FI is an objective way of looking at whether there is a big difference in the number of events between the groups. The FI shows how many patients with a different outcome in one of the groups would make the study stop being statistically significant. A fragility index of 3 indicates that if 3 patients had had a different outcome in the control group, the study would have been negative.

Therefore, a higher FI demonstrates robustness of the data and a lower one shows fragility (1). People tend to think that a big study has the best data, but the truth is, the number of patients enrolled is not as important as 1. the number of events observed and 2. the absolute risk difference between the groups.

The FI adds another dimension the overly simplified frequentist approach to science (p-value). It is helpful to consider in context and when calculated as the Fragility quotient (FI/number of participants in the study) or susceptibility index (loss to follow up - FI/loss to follow up).

Unfortunately, many published studies in high impact journals have low fragility index and even more have a FI lower than the number of patients loss to follow up. (2)

Strengths:

-Allows an objective comparison between studies and gives more confidence when applying data.

-When fragility index is compared to the loss of follow up, it can help determine that the statistical significance might not be as set in stone as we think. If the number lost to follow up is larger than the fragility index it raises concerns for a false positive trial.

Limitations to the fragility index:

-Can only be used with yes / no events. It is harder to calculate with continuous variables or with time-to-event outcomes (Hazard ratios)

-It is correlated inversely with the p-value. The language of strength used can cause confusion as to what p-values really represent. It represents statistical significance, not clinical significance.

-It uses a dichotomous frequentist approach and does not consider the prior probability of the hypothesis being actually true (Bayesian approach)

-There is no standard "cutoff" to use when analyzing studies. Fragility or Robustness is somewhat relative.

1. Dettori JR, Norvell DC. How Fragile Are the Results of a Trial? The Fragility Index. Global Spine J. 2020 Oct;10(7):940-942. doi: 10.1177/2192568220941684. Epub 2020 Jul 15. PMID: 32677531; PMCID: PMC7485073.

2. Ho AK. The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies. J Gen Intern Med. 2022 Jan;37(1):206-211. doi: 10.1007/s11606-021-06999-9. Epub 2021 Aug 6. PMID: 34357573; PMCID: PMC8739402.