1

A fishing expedition is when a scientist initiates a large number of investigations simultaneously on the same data set. A good conceptual example is a study of the effect of a certain environmental chemical on cancer. A researcher could look for an association with bladder cancer, brain cancer, colon cancer, liver cancer, lung cancer, ovarian cancer, prostate cancer, skin cancer, and that funny little thing hanging in the back of your throat cancer. With so many different cancers to look at, the scientist is bound to find something worth publishing, even if the chemical is as harmless as table salt. It's like placing a single bet at a casino, but getting to spin the roulette wheel a couple dozen times.

The Bonferroni correction is a statistical adjustment for the multiple comparisons</strong> that are made in a fishing expedition. It effectively raises the standard of proof needed when a scientist looks at a wide range of hypotheses simultaneously.

The Bonferroni correction is quite simple. If we are testing n outcomes instead of a single outcome, we divide our alpha level by n. Suppose we were looking at the association of sodium chloride and 20 different types of cancers. Instead of testing at the tradition .05 alpha level, we would test at alpha=.05/20=.0025 level. This would ensure that the overall chance of making a Type I error is still less than .05.

You can also apply the Bonferroni correction by adjusting the p-value. A Bonferroni adjusted p-value would just be the normal p-value multiplied by the number of outcomes being tested. If the adjusted p-value ended up greater than 1.0, it would be rounded down to 1.0.

Some scientists dislike the use of the Bonferroni correction; they prefer instead that researchers clearly label any results from a fishing expedition as preliminary and/or exploratory. Furthermore, the Bonferroni correction can cause a substantial loss in the precision of your research findings.

The global null hypothesis

Your general perspective on hypothesis testing is important. One perspective that clearly calls for a Bonferroni adjustment is a global null hypothesis.

Suppose that you are measuring a large number of outcome variables, and you will conclude that a treatment is effective (or that an exposure is dangerous) if you find a statistically significant effect on ANY ONE one of your outcome variables. So a new drug for asthma would be considered effective if it

  1. reduced the number of symptoms as measured by the number of wheezing episodes
  2. OR the number of emergency room visits related to asthma
  3. OR the number of patients who no longer required steroid treatment
  4. OR if the average patient had an improvement in lung capacity as measured by FEV1
  5. OR as measured by the FVC value
  6. OR as measured by the FEV1/FVC ratio
  7. OR as measured by the FEF25-75 value
  8. OR as measured by the PEFR value
  9. OR if the average patient had a better quality of life as measured by the SF-36.

When there is a clear global null hypothesis, then you should use a Bonferroni adjustment.

Consider a restrictive hypothesis, a conceptual hypothesis at the other extreme. Suppose that you will consider a treatment as effective only if it shows a statistically significant improvement in all of those outcome variables.

With a restrictive hypothesis, you might be justified in increasing your alpha level to .10 or .15 since your criteria for success is so stringent.

In truth most studies do not gravitate to either extreme, making it difficult for you to decide whether to use the Bonferroni adjustment.

Designating primary outcome variables

When you need to examine many different outcome measures in a single research study, you still may be able to keep a narrow focus by specifying a small number of your outcome measures as primary variables. Typically, a researcher might specify 3-5 variables as primary. The fewer primary outcome variables, the better. You would then label as secondary those variables not identified as primary outcome variables.

When you designate a small number of primary variables, you are making an implicit decision. The success or failure of your intervention will be judged almost entirely by the primary variables. If you find that none of the primary variables are statistically significant, then you will conclude that the intervention was not successful. You would still discuss any significant findings among your secondary outcome variables, but these findings would be considered tentative and would require replication.

Examining post hoc mechanisms